The Google team provided so little information about its code and how it was tested that the study amounted to nothing more than a promotion of proprietary tech.
“We couldn’t take it anymore,” says Benjamin Haibe-Kains.“When we saw that paper from Google, we realized that it was yet another example of a very high-profile journal publishing a very exciting study that has nothing to do with science,” he says.
Science is built on a bedrock of trust, which typically involves sharing enough details about how research is carried out to enable others to replicate it, verifying results for themselves.
And tech giants carry out more and more research on enormous, expensive clusters of computers that few universities or smaller companies have the resources to access.
“The boundaries between building a product versus doing research are getting fuzzier by the minute,” says Haibe-Kains.
Haibe-Kains would like to see journals like Nature split what they publish into separate streams: reproducible studies on one hand and tech showcases on the other.
On the other hand, new techniques, such as model compression and few-shot learning, could reverse this trend and allow more researchers to work with smaller, more efficient AI.
If it’s done right, that doesn’t have to be a bad thing, says Pineau: “AI is changing the conversation about how industry research labs operate.”
The key will be making sure the wider field gets the chance to participate.