Earlier this year, Kevin Zahnle, a planetary scientist at Nasa’s Ames Research Center, was one of the first people to read a scientific paper that would become the biggest space news of 2020. The study, which Zahnle had been asked to provide comments on while it was being considered for publication in the journal Nature Astronomy made two surprising claims. Firstly, that the authors had spotted signs of a gas called phosphine in Venus’s atmosphere. Secondly, they suggested that this gas might be a sign of life on the uninviting and blisteringly hot planet.

When he read the paper, Zahnle was sceptical. “I can assure you that this reviewer explicitly warned the authors that they were fooling themselves,” he says. The planetary scientist had worries that phosphine signal wasn’t what the study’s authors thought it was. But his main concerns were about the very thing that made the research paper become so important, the reasoning that the presence of phosphine indicated life. On Earth, phosphine is found in small quantities, yet how it is created is up for debate.

Since the 1980s, scientists have theorised that phosphine is created by microbes in oxygen-free environments, like sewage sludge, but it is not a widely-accepted signature of life. Zahnle isn’t convinced that the presence of phosphine on Venus should be interpreted as a possible sign of life – he thinks that it’s more likely that if phosphine is present on the planet, it is probably created by some as-yet-unknown geological phenomenon. The Nature Astronomy paper authors don’t definitively rule out geological processes as the source of the phosphine, but they do conclude that a living entity is the most likely source.

Despite Zahnle’s scepticism, the paper was published. “It’s not unusual to have strong opinions expressed during peer review,” says Paul Byrne, associate professor of planetary science at North Carolina State University, who was not involved in the study. “But I sincerely hope that that reviewer communicated clearly to the team why they felt those researchers were fooling themselves, and that the team defended their methods and conclusions appropriately.”

Jane Greaves from Cardiff University, lead author on the study, does not recall this specific language being used in the review process. It’s possible, she says, the comments were made to the editor and not communicated back to her team. All reviewers on the paper, for which there are usually between two and four for Nature Astronomy, were anonymous until publication. “All of them made their points very clearly, and agreed that the paper was suitable to publish, in the final round of comments,” she says.

Despite his reservations about its conclusions, Zahnle was excited to see the study published. “Publication speeds confirmation or refutation, which becomes the science that stands after the pursuit of glory has been forgotten,” he says.

Now, a month and a half after the initial paper was published, new papers are coming out suggesting the phosphine might not be there at all. In one study, yet to go through the peer review process, researchers led by Ignas Snellen from the Leiden Observatory looked at the data used in the initial research. They analysed it in a different way, and found no evidence for phosphine.

Part of the reason the data analysis was so difficult was that Atacama Large Millimetre Array (ALMA), an array of telescopes, is used to looking at the cold, vast cold clouds in interstellar space and not at nearby Venus, the third brightest natural object in the sky. To make sense of the data it gathered, to calibrate it, and to reduce noise and disturbances, requires a lot of mathematical acrobatics. Greaves and her team fit the data using a 12th order polynomial, a mathematical expression with 12 variables. According to Snellan and his team, using this polynomial actually introduced spurious results. The way they analysed it, no phosphine was found.

So, which method to trust? “No one method is necessarily better than another, or at least not intrinsically more trustworthy,” says Byrne. What’s important, he says, is making sure each method is scrutinised and held to the same standard.

The team behind the original Venus study spotted signs of phosphine from data gathered by the James Clerk Maxwell Telescope (JCMT), then followed it up with a closer look using ALMA. “Any case for arguing that there’s no phosphine in the Venus atmosphere must explain the JCMT detection, too,” says Byrne. The Snellan paper does not explain this, but another one does.

On October 27, a team led by Geronima Villanueva, a planetary astronomer at the Nasa Goddard Space Flight Center, published a paper to the preprint server arXiv. The paper, which is yet to go through review, has been submitted to Nature Astronomy’s Matters Arising section, designed for comments or responses to research published in the journal.

In this comment article, Villanueva and his team argue the phosphine signal has been mixed up with sulphur dioxide, a gas abundant in Venus which produces a signal close to that of phosphine. The researchers say the methods used by Greaves and her team, when analysing both the JCMT and the ALMA data, cannot conclusively differentiate between sulphur dioxide and phosphine.

“The analysis [of the comment article] is sound and measured, and I don’t think many would take issue with their methodology or conclusions,” says Brad Gibson, head of physics and astrophysics at the University of Hull. Initially, the paper had explicitly called for Greaves and her team to retract their study, which Gibson and others did take issue with. But this recommendation was subsequently removed.

Some of the data used in the initial paper is now being reprocessed. According to the researchers who worked on the initial study, something potentially went wrong when processing the data, before delivering it to Greaves and her team.

“The European ALMA Regional Centre Network, who originally calibrated the data that was delivered to Greaves, are now scrutinising it in detail and reprocessing it,” says an ALMA spokesperson. Greaves and her team have said it would be unfair to comment on any papers scrutinising their own results, until the reprocessed data has been published. There is no way of knowing yet how the reprocessing of this data will affect the team’s phosphine detection. But Villanueva and his team refer to the error in their paper, and they say removing it does affect the results.

High profile papers like this one receive more scrutiny than an average scientific paper, but scrutiny itself is not a bad thing. Another astronomer confirmed to WIRED they have just submitted a paper which addresses concerns with the original study, but can’t comment until their own has been through peer review. “The more scrutiny the better,” says Byrne. “If this detection is real, then subsequent observations with different instruments, by different teams, is the best way to assure it.”

The best way to determine whether phosphine is on Venus or not is to go there. “To fly a mission either to the orbit of Venus, or better still, an orbiter and an aerial platform that would both search for that gas, and more broadly characterise the Venus atmosphere, simultaneously,” says Byrne. “We won’t resolve this question fully from Earth.”

We have been there, but a long time ago. In the 1980s, the Russian Vega mission detected a chemical which contained phosphorus in the clouds of Venus. However, the instruments could not determine if it was phosphine. In 1978, Nasa dropped probes into Venus’s atmosphere as part of the Pioneer mission. Rakesh Mogul, professor of biological chemistry at California State Polytechnic University, looked back at the data from this mission, using samples taken between 50km and 60km above the surface. He and his team found evidence for phosphine in the 40-year-old data, although their claims have not been peer reviewed.

But even if we do confirm the presence of phosphine, it doesn’t mean life on Venus. “If phosphine is confirmed beyond doubt to be present at Venus, it’s very unlikely to be biotic in origin,” says Byrne. Other astronomers who, like Byrne, were not involved in the study, agree. Phosphine is not a gas they usually look for when spotting signs of life. That said, any life in such a scorching hot and acidic atmosphere is unlikely to resemble much life we have on Earth.

If it’s not life, working out how phosphine got there could be an exciting task itself. In a lab, heating phosphoric acid to over 200 degrees Celsius can produce phosphine. On Venus, the hottest planet in the solar system, this would be easy, says Zahnle. All it would require would be phosphoric acid, which could be produced from phosphorus trioxide, a molecule that would be stable in Venus’ atmosphere, falling as rain. Exactly how phosphorus trioxide would be produced is a big question, but an exciting one to answer. “The abiotic phosphine cycle would be a very powerful way to interrogate Venus,” says Zahnle.

Publication, scrutiny, gathering more results then going back to the drawing board are all part of the scientific method. This is the incremental nature of science, which doesn’t work in eureka moments. Instead, researchers try their best to interpret the data they have, until new papers with new methods come out and are scrutinised themselves.

“Every method must be thoroughly assessed by peer review and by the broader community to ensure it holds up,” says Byrne. “Those that don’t, we can discard.” It’s too early to discard the work by Greaves and her team yet, but it might also have been too early to celebrate it too.

More great stories from WIRED

🇸🇪 Not every country treated the pandemic the same – did Sweden’s Covid-19 experiment work?

💬 This AI Telegram bot has been abusing thousands of women

🧥 Apple’s new phones have arrived: Should you get the iPhone 12 or iPhone 12 Pro?

🔊 Listen to The WIRED Podcast, the week in science, technology and culture, delivered every Friday

👉 Follow WIRED on Twitter, Instagram, Facebook and LinkedIn