Skip to main content

Research Repository

Advanced Search

Reproducibility in density functional theory calculations of solids

Lejaeghere, Kurt; Bihlmayer, Gustav; Bjoerkman, Torbjoern; Blaha, Peter; Bluegel, Stefan; Blum, Volker; Caliste, Damien; Castelli, Ivano E.; Clark, Stewart J.; Dal Corso, Andrea; de Gironcoli, Stefano; Deutsch, Thierry; Dewhurst, John Kay; Di Marco, Igor; Draxl, Claudia; Dulak, Marcin; Eriksson, Olle; Flores-Livas, Jose A.; Garrity, Kevin F.; Genovese, Luigi; Giannozzi, Paolo; Giantomassi, Matteo; Goedecker, Stefan; Gonze, Xavier; Granaes, Oscar; Gross, E.K.U.; Gulans, Andris; Gygi, Francois; Hamann, D.R.; Hasnip, Phil J.; Holzwarth, N.A.W.; Iusan, Diana; Jochym, Dominik B.; Jollet, Francois; Jones, Daniel; Kresse, Georg; Koepernik, Klaus; Kuecuekbenli, Emine; Kvashnin, Yaroslav O.; Locht, Inka L.M.; Lubeck, Sven; Marsman, Martijn; Marzari, Nicola; Nitzsche, Ulrike; Nordstrom, Lars; Ozaki, Taisuke; Paulatto, Lorenzo; Pickard, Chris J.; Poelmans, Ward; Probert, Matt I.J.; Refson, Keith; Richter, Manuel; Rignanese, Gian-Marco; Saha, Santanu; Scheffler, Matthias; Schlipf, Martin; Schwarz,...

Reproducibility in density functional theory calculations of solids Thumbnail


Kurt Lejaeghere

Gustav Bihlmayer

Torbjoern Bjoerkman

Peter Blaha

Stefan Bluegel

Volker Blum

Damien Caliste

Ivano E. Castelli

Andrea Dal Corso

Stefano de Gironcoli

Thierry Deutsch

John Kay Dewhurst

Igor Di Marco

Claudia Draxl

Marcin Dulak

Olle Eriksson

Jose A. Flores-Livas

Kevin F. Garrity

Luigi Genovese

Paolo Giannozzi

Matteo Giantomassi

Stefan Goedecker

Xavier Gonze

Oscar Granaes

E.K.U. Gross

Andris Gulans

Francois Gygi

D.R. Hamann

Phil J. Hasnip

N.A.W. Holzwarth

Diana Iusan

Dominik B. Jochym

Francois Jollet

Daniel Jones

Georg Kresse

Klaus Koepernik

Emine Kuecuekbenli

Yaroslav O. Kvashnin

Inka L.M. Locht

Sven Lubeck

Martijn Marsman

Nicola Marzari

Ulrike Nitzsche

Lars Nordstrom

Taisuke Ozaki

Lorenzo Paulatto

Chris J. Pickard

Ward Poelmans

Matt I.J. Probert

Keith Refson

Manuel Richter

Gian-Marco Rignanese

Santanu Saha

Matthias Scheffler

Martin Schlipf

Karlheinz Schwarz

Sangeeta Sharma

Francesca Tavazza

Patrik Thunstroem

Alexandre Tkatchenko

Marc Torrent

David Vanderbilt

Michiel J. van Setten

Veronique Van Speybroeck

John M. Wills

Jonathan R. Yates

Guo-Xu Zhang

Stefaan Cottenier


NTRODUCTION The reproducibility of results is one of the underlying principles of science. An observation can only be accepted by the scientific community when it can be confirmed by independent studies. However, reproducibility does not come easily. Recent works have painfully exposed cases where previous conclusions were not upheld. The scrutiny of the scientific community has also turned to research involving computer programs, finding that reproducibility depends more strongly on implementation than commonly thought. These problems are especially relevant for property predictions of crystals and molecules, which hinge on precise computer implementations of the governing equation of quantum physics. RATIONALE This work focuses on density functional theory (DFT), a particularly popular quantum method for both academic and industrial applications. More than 15,000 DFT papers are published each year, and DFT is now increasingly used in an automated fashion to build large databases or apply multiscale techniques with limited human supervision. Therefore, the reproducibility of DFT results underlies the scientific credibility of a substantial fraction of current work in the natural and engineering sciences. A plethora of DFT computer codes are available, many of them differing considerably in their details of implementation, and each yielding a certain “precision” relative to other codes. How is one to decide for more than a few simple cases which code predicts the correct result, and which does not? We devised a procedure to assess the precision of DFT methods and used this to demonstrate reproducibility among many of the most widely used DFT codes. The essential part of this assessment is a pairwise comparison of a wide range of methods with respect to their predictions of the equations of state of the elemental crystals. This effort required the combined expertise of a large group of code developers and expert users. RESULTS We calculated equation-of-state data for four classes of DFT implementations, totaling 40 methods. Most codes agree very well, with pairwise differences that are comparable to those between different high-precision experiments. Even in the case of pseudization approaches, which largely depend on the atomic potentials used, a similar precision can be obtained as when using the full potential. The remaining deviations are due to subtle effects, such as specific numerical implementations or the treatment of relativistic terms. CONCLUSION Our work demonstrates that the precision of DFT implementations can be determined, even in the absence of one absolute reference code. Although this was not the case 5 to 10 years ago, most of the commonly used codes and methods are now found to predict essentially identical results. The established precision of DFT codes not only ensures the reproducibility of DFT predictions but also puts several past and future developments on a firmer footing. Any newly developed methodology can now be tested against the benchmark to verify whether it reaches the same level of precision. New DFT applications can be shown to have used a sufficiently precise method. Moreover, high-precision DFT calculations are essential for developing improvements to DFT methodology, such as new density functionals, which may further increase the predictive power of the simulations.


Lejaeghere, K., Bihlmayer, G., Bjoerkman, T., Blaha, P., Bluegel, S., Blum, V., …Cottenier, S. (2016). Reproducibility in density functional theory calculations of solids. Science, 351(6280), Article aad3000.

Journal Article Type Article
Acceptance Date Feb 19, 2016
Online Publication Date Mar 25, 2016
Publication Date Mar 25, 2016
Deposit Date Nov 15, 2016
Publicly Available Date Feb 24, 2017
Journal Science
Print ISSN 0036-8075
Electronic ISSN 1095-9203
Publisher American Association for the Advancement of Science
Peer Reviewed Peer Reviewed
Volume 351
Issue 6280
Article Number aad3000


Accepted Journal Article (306 Kb)

Copyright Statement
This is the author’s version of the work. It is posted here by permission of the AAAS for personal use, not for redistribution. The definitive version was published in Science on 25 Mar 2016: Vol. 351, Issue 6280, DOI: 10.1126/science.aad3000

You might also like

Downloadable Citations