WASHINGTON \u2014 A NASA investigation blamed millions of dollars in damage to one of its largest Deep Space Network (DSN) antennas on poor training and procedures.<\/p>\n
NASA released this month a redacted version of the final report<\/a> investigating an incident at its DSS-14 antenna, a 70-meter DSN antenna in Goldstone, California. That antenna has been offline since the Sept. 16 incident.<\/p>\n The antenna suffered damage when it over-rotated, stressing and breaking cables and hoses<\/a>, including those carrying water for a fire-suppression system. That flooded the base of the antenna with more than 750,000 liters of water that also contained glycol.<\/p>\n The incident caused between $4.1 million and $4.6 million in damage to the antenna, according to the report, enough to classify the incident as a \u201cType A\u201d mishap by NASA.<\/p>\n The investigation identified six critical events leading up to the mishap, although the public version of the report redacts nearly all details about those events. Those events started an unspecified period before the mishap when a hydraulic limit system for the antenna became inoperable.<\/p>\n There was an anomaly Sept. 15 during a communications session with the Juno spacecraft, according to the report\u2019s timeline, leading to maintenance and troubleshooting. The antenna was then repeatedly driven into rotation limits, causing troubleshooting that led to the antenna being inadvertently over-rotated during another communications session with Juno on Sept. 16. There was additional damage when controllers tried to stow the antenna, causing further over-rotation.<\/p>\n The investigation identified several root causes for the mishap. They included personnel at the site who were not properly trained, procedures that were inadequate and a facility that was \u201coverly reliant on undocumented behaviors and institutional knowledge.\u201d<\/p>\n \u201cThe investigation cited some issues, workplace culture, inadequate procedures, undocumented practices and some unclear roles and responsibilities,\u201d Greg Heckler, deputy program manager for capability development in NASA\u2019s Space Communications and Navigation, or SCaN, program, said during a June 10 meeting of the Small Bodies Assessment Group.<\/p>\n The report highlighted a reliance on what it called \u201cpersonal heroics\u201d by personnel at the Goldstone site, described as being \u201cwilling to do whatever it takes to keep the antenna running.\u201d The problem with that, the report concluded, is that it led them to do tasks outside their qualifications, work extended hours that led to fatigue and skip tests they felt would have delayed returning the antenna to operations.<\/p>\n Had the site\u2019s personnel \u201cacted with greater deliberation or shown more willingness to leave the antenna in a failed state at any point during the mishap,\u201d the report concluded, \u201cthe undesired outcome likely would not have occurred.\u201d<\/p>\n The report provided 20 recommendations, including a need to \u201cincentivize technical rigor over personal heroics\u201d as well as improve training and procedures.<\/p>\n \u201cOur ops team in SCaN is looking across the entirety of the SCaN network, not just the Deep Space Network but also the Near Space Network, given these investigation findings to see if there\u2019s similar behaviors,\u201d Heckler said, \u201cso they can develop a cohesive and comprehensive plan to address that.\u201d<\/p>\n