What does good look like? Whether trying to predict World Cup results or assess the relative merits of emerging technologies, establishing a meaningful set of performance benchmarks is crucial but challenging. Using metrics such as expected goals or possession percentages to assess football teams is not particularly accurate, as a series of shock results at this World Cup show. For nascent technologies such as quantum computing, covered in IDTechEx‘s recent report, establishing universally applicable benchmarks that communicate development timelines and facilitate meaningful comparisons between competing companies and technical approaches is even harder.
Possession Is Not Everything
There are countless ways to measure the quality of a football team: possession, shots on target, expected goals, pass accuracy, aerial challenges won – the list goes on. While each of these metrics has value, ultimately, none can be considered in isolation. The same is true for quantum computers. Popular and widely used benchmarking metrics include qubit number, coherence time, error rate, quantum volume, algorithmic volume, and circuit layer operations per second (see the image below for expanded definitions).
However, unlike goals in football, for quantum computing, there is not a single data point that provides an indisputable measure of quality. Furthermore, many benchmarks used in the quantum computing industry are even more complex to understand than ‘expected goals’, especially for those without a physics background.
Despite these challenges, benchmarking the emerging quantum computing industry is essential for predicting technological trends, making investments, and initiating partnerships. As such, those interested in this potentially transformative technology would benefit from a greater understanding of common benchmarking methods, the trade-offs associated with each approach, and how they can be used in combination. There is also scope for hardware developers to improve in communicating how quantum computing benchmarks can be used effectively when engaging with non-expert stakeholders.
Biased Referees
One of the challenges of developing benchmarking metrics for a new technology is that there are no existing rules. As a result, some quantum computing benchmarking metrics have been developed by companies building quantum computers themselves. This risks the equivalent of combining football manager and referee into a single role – we do not want teams that control the ball deciding to use ‘possession percentage’ to decide who won the match.
While a completely independent set of benchmarks would clearly be desirable, many of those capable of understanding the technology and setting meaningful benchmarks work for companies developing quantum computers. Furthermore, developing benchmarking metrics in-house that are then widely adopted offers a substantial strategic advantage – as the saying goes, ‘those who define the terms, control the debate’.
While some internally developed benchmarks have done a good job of consolidating existing metrics into a single parameter, they risk introducing yet more terminology, thus further fragmenting the benchmarking ecosystem. There is, therefore, a considerable need for independent, widely applicable benchmarks that facilitate comparison regarding the status of quantum computing companies and technologies towards commercialization.
Focus on the Next Game
Technology readiness levels (TRL) are widely used across many sectors to assess the status of emerging technologies. However, they are not well suited to computing since each commercially available iteration of a computing technology demonstrates a substantial performance improvement despite always having a TRL of 10.
This consideration of ‘continuous computing’ has been missed by some existing efforts to benchmark quantum computers. For example, some benchmarking scales end at the production of a ‘universal machine’ or demonstration of ‘supremacy’. If this logic were applied to the classical computing industry, benchmarks would not show any progress since the 1980s. Rather than focusing on TRL, the focus for quantum computers should be on continuously expanding capabilities to serve an ever-growing addressable market. The question should be, ‘are quantum computers ready yet’ but instead, ‘who else are they useful for now?’
It Is All About the Silverware
Despite the wide variety of statistical benchmarks, the ultimate measure of football success is winning trophies. In quantum computing, satisfying a demanding array of benchmarks is only impressive if it ultimately provides value to end-users. This overarching goal can be obscured by the focus on technical benchmarks and ideas such as ‘supremacy’, ‘million-qubit milestones’, and ‘fault tolerance’. The key tipping point will arrive when quantum hardware has been sufficiently scaled up to tackle quantum algorithms that can solve commercially valuable problems, such as drug development or weather forecasting, that cannot be achieved by other means. Measuring how quantum computing companies are progressing relative to this aim and the range of valuable problems that can be addressed is arguably the most important requirement for a universal benchmarking metric.
Quantum Commercial Readiness Level
To help evaluate the status of quantum computing companies and technologies on this road to providing significant societal benefits, IDTechEx has developed an unbiased ‘Quantum Commercial Readiness Level’ (QCRL) index. The QCRL ranges from 1-10, with stages 1-5 tracking use cases from educational tools to versatile machines and 6-10 assessing scalability from cloud access to personal ownership. The aspiration is that it will remain relevant as the industry progresses beyond ‘quantum supremacy’, is hardware agnostic, and easy to understand.
Outlook
There are many challenges when it comes to benchmarking quantum computers. The terminology and science underpinning them are complex and there are many competing technologies. Companies developing measures of good quantum computers are also using them to rate their own technology, and familiar standards like technology readiness level are not very useful. However, benchmarking is so essential to demonstrate progress to investors and other stakeholders that efforts to standardize it for the industry must continue. An emphasis on unbiased input and how close the technology is to adding commercial/societal value is crucial. In short, what this game needs is a goal.
More details and twenty-year market forecasts can be found in IDTechEx’s comprehensive report “Quantum Computing 2023-2043“. This includes individual forecast lines for eight different technology categories, including superconducting, photonic, trapped-ion, neutral atom, silicon spin, topological, diamond defect, and annealers. There are also 60-year projections covering meta-trends for quantum computer adoption, going beyond the horizon of a realized versatile computer, and looking ahead to mass-market adoption.
To find out more, including downloadable sample pages, please visit www.IDTechEx.com/QuantumComputing.
