Narrative:

While working oceanic position xyz at approximately XA30, I received several system messages advising me of an odaps system startover. Prior to this, I had climbed air carrier X from FL330 to FL350. When I received the level report on this aircraft, the message showed a red atl for air carrier X of FL350. When I looked at the isd, it showed air carrier X at FL330. The database was no longer reflecting the altitude change, even though it had accepted the change when I cleared the aircraft. I immediately did a manual database altitude change to FL350 on air carrier X. Odaps accepted the change. I then attempted to show the aircraft level with no success. Upon the next startover, the FL350 was again deleted. Finally, after several attempts, I was able to insert the new altitude and show air carrier X level in the database. Additionally, I was unable to climb an odl aircraft (per his request) due to a 'mc frozen' message. I also received no less than 3 messages stating that air carrier Y was an unsuccessful flight plan pass to ZLA. This happened during the busy traffic period for ZOA oceanic traffic. The other ZOA oceanic sectors were experiencing similar sits. At approximately XB15, a complete 'cold start' was required, which included a total dump of all flight plans. After the cold start, staffing in the area was affected, as several oceanic controllers were required to reinput the lost flight plans. This had an affect on the transition radar sectors, as they were now being required to give incoming aircraft full route clrncs to their destination airport (due to uncertainty of the correctness of the flight plan). This also affected oceanic sectors getting their flight strips in a timely manner, as strips were being generated 25 times or more on the same aircraft. Air carrier Z is an example of this, going from honolulu to manila, generating at least 10 strips each time (total of 250 or more worthless strips), causing delays on other active strips to be delivered. During this time, controllers did not climb aircraft due to the uncertainty/instability of the system. This impacted fuel costs to the airlines. Odaps outages, both scheduled and unscheduled, are a recurring problem at ZOA. It is my understanding that the NAS/nom in charge was unable to contact any aos personnel during this particular outage (possibly due to them being en route to the facility). Additionally, it appeared that the NAS/nom did not seem to possess requisite knowledge to correct problems (my opinion). He advised the area supervisor that the cold start would only take 5 mins and that odl would be lost. This was not the case.

Google
 

Original NASA ASRS Text

Title: ZOA OCEANIC MANUAL CTLR INCURS ODAPS SOFTWARE INTERRUPT REQUIRING A SYS 'COLD START.' EXCESSIVE MANUAL STRIP PRODUCTION A RESULT.

Narrative: WHILE WORKING OCEANIC POS XYZ AT APPROX XA30, I RECEIVED SEVERAL SYS MESSAGES ADVISING ME OF AN ODAPS SYS STARTOVER. PRIOR TO THIS, I HAD CLBED ACR X FROM FL330 TO FL350. WHEN I RECEIVED THE LEVEL RPT ON THIS ACFT, THE MESSAGE SHOWED A RED ATL FOR ACR X OF FL350. WHEN I LOOKED AT THE ISD, IT SHOWED ACR X AT FL330. THE DATABASE WAS NO LONGER REFLECTING THE ALT CHANGE, EVEN THOUGH IT HAD ACCEPTED THE CHANGE WHEN I CLRED THE ACFT. I IMMEDIATELY DID A MANUAL DATABASE ALT CHANGE TO FL350 ON ACR X. ODAPS ACCEPTED THE CHANGE. I THEN ATTEMPTED TO SHOW THE ACFT LEVEL WITH NO SUCCESS. UPON THE NEXT STARTOVER, THE FL350 WAS AGAIN DELETED. FINALLY, AFTER SEVERAL ATTEMPTS, I WAS ABLE TO INSERT THE NEW ALT AND SHOW ACR X LEVEL IN THE DATABASE. ADDITIONALLY, I WAS UNABLE TO CLB AN ODL ACFT (PER HIS REQUEST) DUE TO A 'MC FROZEN' MESSAGE. I ALSO RECEIVED NO LESS THAN 3 MESSAGES STATING THAT ACR Y WAS AN UNSUCCESSFUL FLT PLAN PASS TO ZLA. THIS HAPPENED DURING THE BUSY TFC PERIOD FOR ZOA OCEANIC TFC. THE OTHER ZOA OCEANIC SECTORS WERE EXPERIENCING SIMILAR SITS. AT APPROX XB15, A COMPLETE 'COLD START' WAS REQUIRED, WHICH INCLUDED A TOTAL DUMP OF ALL FLT PLANS. AFTER THE COLD START, STAFFING IN THE AREA WAS AFFECTED, AS SEVERAL OCEANIC CTLRS WERE REQUIRED TO REINPUT THE LOST FLT PLANS. THIS HAD AN AFFECT ON THE TRANSITION RADAR SECTORS, AS THEY WERE NOW BEING REQUIRED TO GIVE INCOMING ACFT FULL RTE CLRNCS TO THEIR DEST ARPT (DUE TO UNCERTAINTY OF THE CORRECTNESS OF THE FLT PLAN). THIS ALSO AFFECTED OCEANIC SECTORS GETTING THEIR FLT STRIPS IN A TIMELY MANNER, AS STRIPS WERE BEING GENERATED 25 TIMES OR MORE ON THE SAME ACFT. ACR Z IS AN EXAMPLE OF THIS, GOING FROM HONOLULU TO MANILA, GENERATING AT LEAST 10 STRIPS EACH TIME (TOTAL OF 250 OR MORE WORTHLESS STRIPS), CAUSING DELAYS ON OTHER ACTIVE STRIPS TO BE DELIVERED. DURING THIS TIME, CTLRS DID NOT CLB ACFT DUE TO THE UNCERTAINTY/INSTABILITY OF THE SYS. THIS IMPACTED FUEL COSTS TO THE AIRLINES. ODAPS OUTAGES, BOTH SCHEDULED AND UNSCHEDULED, ARE A RECURRING PROB AT ZOA. IT IS MY UNDERSTANDING THAT THE NAS/NOM IN CHARGE WAS UNABLE TO CONTACT ANY AOS PERSONNEL DURING THIS PARTICULAR OUTAGE (POSSIBLY DUE TO THEM BEING ENRTE TO THE FACILITY). ADDITIONALLY, IT APPEARED THAT THE NAS/NOM DID NOT SEEM TO POSSESS REQUISITE KNOWLEDGE TO CORRECT PROBS (MY OPINION). HE ADVISED THE AREA SUPVR THAT THE COLD START WOULD ONLY TAKE 5 MINS AND THAT ODL WOULD BE LOST. THIS WAS NOT THE CASE.

Data retrieved from NASA's ASRS site as of July 2007 and automatically converted to unabbreviated mixed upper/lowercase text. This report is for informational purposes with no guarantee of accuracy. See NASA's ASRS site for official report.