When Machines Search For Discoveries, Science Has To Count The Search – OpEd
The 750 GeV LHC photon excess illustrated the classic “look-elsewhere effect”: a locally impressive anomaly can lose significance once the full breadth of a search is accounted for.
Adaptive AI pipelines amplify this problem because they generate and prune hypotheses during the search, making the true multiplicity hard to see if only the winning result is reported.
Robust AI-assisted discovery requires logging the full search trace, calibrating against realistic null replays, freezing rules before final confirmation, and disclosing search history so machine-scale exploration remains scientifically accountable.
AI systems can scan millions of hypotheses before surfacing one anomaly. The statistical evidence must account for the search that produced it.
In late 2015, physicists at CERN found something irresistible in the new data from the Large Hadron Collider: an excess of photon pairs near 750 GeV, where no known particle was expected. ATLAS reported a local significance approaching four sigma for one interpretation of the bump. CMS saw a smaller compatible excess. For a few months, the possibility of a new particle became one of the hottest questions in high-energy physics.
Then the 2016 data arrived. The bump did not. CERN concluded that the much larger data set had failed to reproduce the excess and that it appeared to have been a statistical fluctuation.
The episode was a vivid lesson in a problem particle physicists know well: the look-elsewhere effect. A peak can look impressive at one location while becoming much less surprising once you account for all the places a search could have found a peak. The local question is, “How unusual is this result here?” The global question is, “How often would a search this broad find something at least this unusual somewhere?” Those are different probability questions. In the 750 GeV case the gap between them was public at the time. Against its local figure approaching four sigma, ATLAS reported a global significance of roughly 2.3 sigma once the range of masses the search covered was........
