Engineering Lessons From The Super-Kamiokande Neutrino Observatory Failure – Hackaday
Every engineer is going to have a bad day, but only an unlucky few will have a day so bad that it registers on a seismometer.We’ve always had a morbid fascination with engineering mega-failures, few of which escape our attention. But we’d never heard of the Super-Kamiokande neutrino detector implosion until stumbling upon [Alexander the OK]’s video of the 2001 event. The first half of the video below describes neutrinos in some detail and the engineering problems related to detecting and studying a particle so elusive that it can pass through the entire planet without hitting anything. The Super-Kamiokande detector was built to solve that problem, courtesy of an enormous tank of ultrapure water buried 1,000 meters inside a mountain in Japan and lined with over 10,000 supersized photomultiplier tubes to detect the faint pulses of Chernkov radiation emitted on the rare occasion that a neutrino interacts with a water molecule.Those enormous PM tubes would be the trigger for the sudden demise of the Super-K , which is covered in the second half of the video. During operations to refill the observatory after routine maintenance, technicians noticed a bang followed by a crescendo of noise from the thirteen-story-tall tank. They quickly powered down the system and took a look inside the tank to find almost every PM tube destroyed. The resulting investigation revealed that the tubes had failed in sequence following the sudden implosion of a single tube at the bottom of the tank. That implosion caused a shock wave to propagate through the water to surrounding tubes which exceeded their design limits, causing further implosions and further destruction. The cascading implosion took a full ten seconds to finish its wave of destruction, which destroyed $7 million worth of tubes.The interesting part about this is the root cause analysis, which boils down to the fact that you shouldn’t stand on 50-cm photomultiplier tubes. Also at fault was the testing regimen for the tubes, which the project engineers anticipated could cause a cascading implosion. They tested this but were unable to cause a cascade failure, leading them to the conclusion it wasn’t likely to happen. But analysis of the destruction revealed a flaw in the testing, which should give pause to anyone who ever had to design a test like this before.Luckily, nobody was killed or even hurt during the Super-K incident. The observatory was repaired with upgraded tubes and remains in service to this day, with an even bigger Hyper-Kamiokande detector in the works. We’ve covered neutrino observatories before, so check that out if you want more background on the science.Insert “series of tubes” and “they should’ve used transistors” jokes here.—
On a serious note, high-cost engineering failures like this need to be studied so they don’t happen again.Someone should write a book perhaps.https://www.amazon.com/gp/product/B0B3HS4C98/Looks like they studied it very carefully (and they did testing to discover the exact same phenomenon beforehand, but they unfortunately did it in insufficiently deep water). The new detectors have some kind of acrylic screens inside to dampen implosions, I suspect a bit like the safety devices inside CRTs once they were mature enough.The new hyper-kamiokande has far more depth and volume, hopefully their precautions were adequate and it works without a hitch“The new detectors have some kind of acrylic screens inside”Outside. It’s basically a “sleeve” around the PMT, with a fiber-reinforced bottom and acrylic top. Thankfully the amount of light received in SuperK is enough that the small amount of loss in the acrylic (few percent) wasn’t a big deal.You can see a picture of what it looks like here:https://cerncourier.com/a/superkamiokande-to-be-rebuilt-this-year/“high-cost engineering failures like this need to be studied so they don’t happen again.”They’ll always happen. People just make mistakes. They won’t make the same mistake again, but it’ll be a New and Interesting Mistake. Studying them is really important, but the problem is usually that the people who make the new mistakes are the ones who didn’t study the old ones. And of course a lot of times the fixes to the old mistakes cause new mistakes because there was too much focus on the old mistake.Sigh.On a more direct note as someone who encountered quite a few people from Super-K at conferences very shortly after this happened I can say that there were a lot of somber tones during those talks.Henry Petrowski wrote a book about this, “To Engineer Is Human”, which describes the cycle of building something new, making it reliable, optimizing it, finding a failure as a result of optimization, and starting again, and ways in which engineers can design the optimization problem to reduce lethality on failure. He was largely writing about civil engineering, but the same thing holds, usually with reduced lethality, for most other engineering disciplines.Aren’t PM tubes supposed to use cesium? Isn’t cesium supposed to react vigorously with water?The amount of photocathode material (which consists of various alkali elements since they need to be low work function and the alkalis love to give up electrons) is just amazingly tiny.If it wasn’t so depressing when a PMT cracks (goodbye, a few thousand dollars) it’d be cool – the colored tinge on it from the photocathode just disappears almost instantly as soon as you lose vacuum.Thankfully the most common failure points are where the leads stick through so the whole thing doesn’t usually go “boom” like what happened here. PMTs in general are surprisingly strong – I sprinted away in terror after someone’s hand slipped and an 8″ one fell to a concrete floor… only to watch it bounce, totally fine – but the lead penetrations (where you’ve got dissimilar materials touching) and neck bend are susceptible to cracks. We had one manufacturer not believe us that their design was failing often until we sent them back a slew of failed PMTs all with tiny cracks at the leads.A cascade failure (shockwave in water)
Of cascade sensors (photomultipliers)
IN A CASCADE DETECTOR!? (radiation)Um, I think the cascade in photomultipliers is the same as the one in a cascade detector. But good try.yo dawg…… I put some cascade failures in your cascade failures.… to cascade the failures into your amazing cascading cascade failures failures.There is a neutrino “cascade event”, but I’m not sure if that has any relation to the “cascade detector” name https://icecube.wisc.edu/news/research/2019/11/what-can-cascade-events-tell-us-about-neutrino-sources/I’m not sure where the “cascade detector” bit came from, but the higher-energy neutrinos SuperK was looking at came from atmospheric cascades (in the field, they’re “air showers”) from cosmic rays.It’s a little funny because the video mentions this since it needs to be deep underground to avoid muons from those air showers, but when those muons decay, they decay into electrons and a muon neutrino, and that neutrino basically doesn’t care how deep you are.What SuperK won the Nobel Prize for was observing that there were significantly fewer of those neutrinos (“atmospheric neutrinos”, specifically muon neutrinos here) coming through the Earth or from the side than you’d expect. That, in combination with other experiments, led to the conclusion that the neutrinos created in one type of interaction (say, when a muon decays) don’t stay as that ‘type’ of interaction – that’s usually called “neutrino oscillation” as in “muon neutrinos oscillate into tau neutrinos,” although in truth it’s really “muon neutrinos aren’t actually real things that propagate – the things that propagate can interact in multiple ways,” which leads to the conclusion that there’s a mass difference between those propagating states.The nitpickingly-correct version is then:a cascade failure (many failures from one failure)
of cascade sensors (many electrons from one photon)
in a cascade detector (many neutrinos from one cosmic ray)Nope, because the photomultiplier is detecting the light from the cascade of the particles, e.g. muons, cosmic rays, etc…Maybe I missed it watching the video, but didn’t see anything about how the tubes and electronics were isolated and connected, there’s alot of pressure at the bottom of the tank. Also, the system design allowed a tube to fail and still function with exposed flooded elements.The electronics were all outside the detector tanks, they just routed cables to each of the PMTs. Lots, and lots, and lots of cables.The tubes themselves were installed in modules of 12 20″ PMTs (plus two smaller PMTs looking outward) with a support frame tiled about the detector. The paper with design details isn’t freely accessible, I think, but this image link should (?) work?https://ars.els-cdn.com/content/image/1-s2.0-S016890020300425X-gr11.jpgdumb question: why doesn’t the water just fill the vacuum with no fuss?it does, but then it hits the other end of the vacuum with all the energy gained by the pressure differenceThat question isn’t dumb. The problem is: imagine the water is su led into the tube from all sides and due to the tremendous water pressure filling the vacuum is accelerated to extreme speeds. All the water in the tank follows as filling the void doesn’t leave a hole anywhere else than at the surface. And then the tube is filled. Water at extremely high speeds collides with water at extremely high speed. All the water that followed this water is still moving, and much water even at low speed has the energy of a freight train. In fact the mathematical equation that explain how electrical grids deal with lightning strikes were created in the palace of Versailles when someone closed a tap and with that abruptly tried to stop 100m of flowing water in the pipes leading to that tap breaking them.
The problem is: all of the energy and momentum from these incidents doesn’t just disappear when the vacuum tube is filled. Your water now is loaded with the force a major explosion would have and vacuum-filled glass tubes.I want an audio recording of that incident.Please be kind and respectful to help make the comments section excellent. (Comment Policy)This site uses Akismet to reduce spam. Learn how your comment data is processed.
By using our website and services, you expressly agree to the placement of our performance, functionality and advertising cookies. Learn more
