Of nearly 4,000 clinical prediction studies that an international team reviewed, about one in eight reported sharing code. Researchers at MIT and Dana-Farber Cancer Institute led the review, following links from papers to hundreds of code repositories. Some authors organized their files carefully and explained them in detail. Others offered almost no guidance, said first author Thomas Sounack.

Clinical prediction models estimate outcomes such as a patient’s risk of developing a disease. When those estimates inform care, it matters how researchers built and tested the model. Access to the data and code lets others check which patients the researchers included, how they handled missing values and how they evaluated the results. Such choices can affect the findings, said Tom Pollard, an MIT research scientist and senior author, but a methods section can describe them only in so much detail.

Tracing code from papers to repositories

The review, published in Nature Medicine, covered studies that developed, updated or validated a prediction model. To qualify, a paper had to cite TRIPOD or TRIPOD+AI, two checklists for reporting how researchers build and evaluate these models. Its full text also had to be available in PubMed Central’s open-access collection.

Authors described and linked to their code in different ways. A large language model helped the team screen papers, find code-sharing statements and extract links. Separate software retrieved the repositories. Pollard said the models helped the team work through information scattered across papers and files. The team assessed how thoroughly authors documented their code, what software it required and on what terms others could reuse it.

Two researchers independently examined 500 papers and a separate sample of repositories. They reconciled their differences before comparing their judgments with the automated results. Agreement was closer on whether a license file existed than on whether the documentation was adequate. Given the possibility of classification errors, the authors put more weight on broad patterns than on exact counts.

What makes shared code usable

Sharing practices differed across countries and journals. The proportion of studies reporting shared code ranged from zero in some journals to over 70 percent in others. Some journals near the low end already had written code-sharing policies, suggesting that a policy alone may not persuade authors to share.

Even authors who make their code public have little guidance on how to organize and explain it. For Sounack, the test is whether someone else can understand the code and run it to reproduce the paper’s results. Some authors supplied synthetic datasets so others could use the code without access to confidential patient records.

The findings are informing TRIPOD-Code, a reporting guideline that builds on the two existing checklists. Clinicians, statisticians and computer scientists all contribute to prediction research and may have different expectations for documentation. Through expert review, the team is seeking agreement on what authors need to include with their analytical code. The aim, Pollard said, is to give researchers practical direction and help journals tell authors what they expect.