Automating Launch Operations

A participant in the U.S. space program likened the Apollo-Saturn to the ancient Tower of Babel. The moon rocket might have duplicated the chaos that marked that earlier dream. In a manual checkout of the Apollo- Saturn’s many systems, hundreds of technicians would have swarmed on and around the space vehicle. Their reports flowing into a central control room would have indeed been a babel. Automated checkout equipment avoided this confusion. The Saturn ground computer checkout system tested 2,700 Saturn functions and its computers monitored 150,000 signals per minute. Acceptance checkout equipment accomplished a similar task for the Apollo spacecraft. Wernher von Braun, after the first successful flight of an Apollo-Saturn V, credited success “to our automatic checkout procedure.”1 The story of Apollo is a study in automation.

Origins of Saturn Automated Checkout

Although automation has no precise meaning that is generally accepted in technical circles, there was considerable automation - however defined in the missile programs of the 1950s. Engineers employed pressure gauges, temperature gauges, frequency detectors, and other devices to passively sequence a series of events. Using relay logic, if event A occurred, then event B took place, and so on. Interlocking circuitry and relay logic allowed ground support equipment to control portions of a countdown [for the SA-1 countdown, see chapter 3-6]. In this chapter, the term automation will imply the use of digital computers and associated equipment.

In checking out the first Saturns, hundreds of control room switches sent signals over electrical lines to test points on the rocket. The launch vehicle responses, returning over the same wires or radio telemetry links, registered on strip charts and meters. The launch team then evaluated the test data. Automation began to change this procedure when, in 1960, Marshall engineers decided to design a test capability for Saturn’s digital guidance computer (its maiden flight would come on SA-5). The first, tentative steps were to parallel, not replace, manual checkout. Quality Division representatives sought a flexible program that could be expanded to include other tests as automation proved itself.2

In early September Debus asked to have the Launch Operations Directorate participate in automated checkout discussions. Before the end of the year two computer systems were under study at Marshall. The Reliability Assurance Laboratory was testing a Packard Bell 250 for factory acceptance checkout while the Guidance and Control Division and the Quality Division investigated the use of an RCA 110 for launch tests. The RCA 110 was among the first priority-interrupt computers on the market. This feature provided for the division of the computer program into several sections, each one having an assigned priority level. The priority interrupt allowed an engineer to switch immediately from one test to another during operations. On the RCA 110 there were eight available levels and the computer could switch to a higher priority test in 100 microseconds. As the early use of the 110 indicated trouble-free operations, it became the workhorse of the Saturn checkout at the Cape.3

Marshall’s first automation plan, published in September 1961, asked the question: “Why automation?” The advantages were speed and accuracy. The author, Ludie Richard, noted that man is a poor test conductor. He cannot run thousands of tests with uniform precision, and he frequently fails to observe the results. Machines ensure standardized testing and an accurate recording of the responses. Further, an automated operation required only a fraction of the time used in a manual procedure. The time savings would permit more testing, an important factor to operations personnel, particularly just prior to launch. With automation Marshall could duplicate the exact conditions under which a failure had occurred. Data would be available at the point of failure to aid in trouble-shooting and fault isolation. Richard also listed some disadvantages. Automated checkout procedures would complicate the Saturn, although the problem could be minimized by designing the automated test system into the vehicle. Another drawback was a lack of user confidence in the system. Richard attributed this to poor planning, either in training the users or in faulty machine language. A long-range problem involved the operators’ possible loss of familiarity with the launch vehicle. Automation might work so well that its users would lose their “feel” of the rocket, with a corresponding drop in their ability to meet a crisis.4

Richard’s automation plan proposed to phase the RCA 110 into Cape operations with the SA-5 launch from LC-37. The blockhouse computer would parallel the launch complex circuitry so that operations could proceed manually if necessary. At first the RCA 110 would check the digital flight computer and monitor other electrical systems. It was hoped that by SA-111 (the first Saturn I flight after the ten R&D launches) equipment reliability and user confidence would permit a fully automated launch.5

Planning for automation accelerated during the following months as Marshall moved from the C-2 to the C-5 version of the advanced Saturn. The Saturn V’s size and LC-39’s greater distance - 4.8 kilometers - from launch control center to pad precluded a manual checkout. On 1 October 1961, von Braun established an Automation Board at Marshall to automate the Saturn V checkout. Thereafter, design of a computer checkout system paralleled launch vehicle development.6

Saturn I-IB Computer Complex

The development of the RCA 110 hardware for Saturn I tests proceeded at Huntsville under the direction of the Astrionics Laboratory with the collaboration of the electrical network group. Richard Jenke’s automation team at KSC furnished operational requirements and participated in a number of design decisions. Some agreements between developer and future user came slowly. The two groups discussed the matter of control panel switches for months; the uncertainty centered on the desired status of the various test devices when the operations switched from a computer program to manual control. Eventually the two groups decided to treat each test device separately and modify the RCA 110 programs when experience dictated a change. About the same time the automation group approved a three-position switch for the Saturn IB control panel. During IB operations all signals, manual as well as automatic, would process through the computer. In the three-position switch, OFF manually terminated a function, such as opening a valve; ON manually initiated a function; and AUTO placed the computer in control for automatic testing. Experience with SA-5 demonstrated the need to prevent the computer from sending any command signals while it monitored the operation. The automation group added a discrete inhibit switch on SA-6 for the remaining Saturn I launches.7

The Cape played a larger role in the development of computer programs, called software. Marshall recognized the launch team’s need to prepare its own tests and allowed KSC to manipulate “Boss,” the 110’s executive control program. Jenke’s group, assisted by RCA, IBM, and Chrysler personnel, combined the 20 manual routines of the guidance computer checkout into four test sequences. While the performance of the RCA 110 on SA-5 left a few skeptics unconvinced, launch officials labeled the computer a success. On the following launch (SA-6) Jenke added a test sequence for automatic azimuth laying. The launch team added a cathode ray tube console (a television screen that displayed alpha-numeric characters received from the computer) for SA-7. The RCA 110 increased its monitoring role during the last three Saturn I launches, the Pegasus series.8

During the Saturn I program, automation moved forward at a slow, deliberate pace; at any time the launch team could have reverted to a manual operation. By the time of the first Saturn IB launch in February 1966, however, KSC was firmly committed to automated testing. While a completely automated checkout was still a long way off, the RCA 110A computer (a 110 with increased memory) was “on line” for the first IB operation. All test transmissions then went through the computer; if it failed, the entire checkout would stop. On-line status represented the decision to use the Saturn IB missions as a testbed for Saturn V automation.

LC-34/37 Computer system schematic
LC-34/37 computer system schematic. From B. E. Duran, “Saturn I/IB Launch Vehicle Operational Status and Experience,” pamphlet 680739, Aeronautic and Space Engineering and Manufacturing Convention, Los Angeles, 7-11 October 1968. Courtesy of the Society of Automotive Engineers, Warrendale, PA.

Two RCA 110As - a “master” computer in the LC-34 blockhouse and a “slave” computer in the automatic ground control station-provided the brains for the Saturn checkout system. A high-speed data link, a coaxial cable running through the LC-34 cableway, connected the two computers. Engineers could initiate launch vehicle tests from display consoles or from programs stored within the master RCA 110A. In either case the computer in the launch control center digitized commands for transmission to the ground control station. The slave computer, housed beneath the umbilical tower, interfaced with the launch vehicle, issuing commands and receiving responses. Both computers were also tied into the spacecraft checkout system.9

In its design, the digital data acquisition system was typical of the digital systems employed at KSC. Sections, or modules, within each computer performed specific functions. The stage module had three distinct elements, each dedicated to one of the launch vehicle elements. Aboard the Saturn IB, three digital data transmitters (for the two stages and the instrument unit) multiplexed the pulse code modulated data, giving each signal a specific time slot on its channel. This data, reflecting the condition of the launch vehicle, was transmitted to ground receiving stations over coaxial cable or radio. The receiver decommutated, i.e., divided the data into its constituent parts, and then conditioned the signals for transmission. Another part of the digital data system, a high-speed memory core, stored the data for use by the 110A computers.10

A digital events evaluator determined whether or not a return signal indicated a change in the launch vehicle’s status. After receiving data from the slave computer, the evaluator compared this signal with pre- programmed information in its memory or with a previous scan of the same function. The event was then time-tagged for identification and the results either printed for display or stored for retrieval by the master RCA 110A at a later date. The evaluator logged every event within approximately two milliseconds, providing real-time - virtually immediate - printouts of event changes.11

Although the Saturn checkout complex was usually identified by the RCA 110A computers, the related electrical support equipment was the larger part of the system. Included was test equipment and a complex distribution system. The RCA 110As relied on input-output address lines to place test equipment in a receive or transmit condition, set sequence control relays, select analog or digital signal lines, drive digital-to-analog converters, and issue warnings. Input-output sense lines informed the computers of test equipment status. The computers employed input-output buffer registers to handle the heavy data flow with the Saturn guidance computer and telemetry system. Hardwire connections tied the 110As in with LC-34’s timing system: the countdown clock, a Greenwich Mean Time clock, and interval timers.12

Saturn V Computer Complex

Dual computers on LC-34 were largely redundant since the distance from launch vehicle to blockhouse was not great enough to attenuate analog signals. On LC-39, however, the dual arrangement was essential. The computer could not accurately process analog signals from the launch vehicle five kilometers away. In this regard, the earlier installation was intended as a testbed for the Saturn V complex. There were also important differences between the two installations. The slave computer in LC-39 was housed on the mobile launcher. Dual data links, running beneath the crawlerway, tied it to the 110A computer in the launch control center. Saturn IB operations were delayed occasionally because test engineers could not reach the slave computer’s peripheral equipment during hazardous pad activities. On LC-39, magnetic tapes and other related equipment were moved back from the mobile launcher to the control center, to be available at all times. Another change involved the display systems. At LC-34, the master computer in the blockhouse processed display signals to the various consoles. LC-39 included a Sanders display system with its own digital computer. The greater size of the Saturn V dictated other changes; for example, LC-39’s checkout system could issue and receive twice as many discrete signals. Perhaps the biggest difference was the increased capacity for testing. In effect there were three of everything: computers for each of the equipped firing rooms, computers in the mobile launchers for interfacing with the launch vehicles, digital data systems, and all the peripheral equipment. As a result, KSC could conduct automated tests on three Saturn V vehicles simultaneously.

As on LC-34, the 110A computers were central elements. The mobile launcher’s computer tied in with the Saturn V vehicle, its peripheral equipment (line printer, card reader-punch, magnetic tapes), the launch control center computer via the data link, and the range clock system. Commands to the launch vehicle went through a discrete output system which employed triple-modular redundancy to minimize errors. Approximately 2,000 test responses returned from the Saturn V. A remote control capability in the control center allowed engineers to continue Saturn tests in the event of a computer breakdown on the mobile launcher. The RCA 110As in the control center had even more interfaces; their data channels controlled signals to and from peripheral equipment, control consoles, the dual data links to the pad, digital data systems of the launch vehicle, computer display systems, the spacecraft computer system, and the countdown clock system.13

The control center was the focal point of Saturn checkout. While the original design included four firing rooms, only three were fully equipped. The various control consoles and display devices were physically grouped by stages or function. Management officials, including the launch director, test supervisor, and test conductors, occupied the first three rows. Within each stage area, test personnel were organized according to functional subsystems. Thus, in the instrument unit area were consoles for the emergency detection system, networks, guidance, stabilization, flight control, and measuring and telemetry. During prelaunch testing, approximately 400 people occupied stations in the firing room. A lesser number manned a backup control room. The test supervisor and conductors directed the operation by means of test procedures, countdown clock readouts, and an intercommunication system.14

Although equipment in a firing room varied with launch vehicle requirements, about 400 consoles were employed. Of these, approximately 100 were cathode ray tube displays. Four large overhead screens provided an additional means of displaying information. Presentations on the 2 × 3-meter translucent screens normally paralleled the test procedures, but the sequence could be altered to display data on a particular launch problem. Information from various sources appeared on the screen - telemetered data, closed-circuit and commercial television, slides, and viewgraphs. The data kept NASA managers and test conductors abreast of the vehicle status.15

Launch control center
An operation underway in the launch control center, firing room 1, March 1967.

An important addition to firing room equipment came after the start of Saturn V operations. While systems engineers could monitor vehicle outputs, the RCA 110A lacked the means to provide full coverage of launch vehicle measurements. Saturn engineers added an alert monitor capability to the launch vehicle display system. The equipment was first tried out in the backup firing room. In 1970 it became an integral part of the control center’s display equipment. The alert monitor system - ten dual sets of cathode ray tube displays tied in with the digital data acquisition system - automatically indicated when certain measurements were out of tolerance.16

The Transition to Automation

With the RCA 110A computers on line for the first Saturn IB mission, Hans Gruene’s Launch Vehicle Operations team found itself totally dependent on computers and computer programs. They were not dependent, however, on automation. Having approached the idea of automated testing with reservations, they insisted that the Saturn design provide for manual testing as well as automation. Hence, LC-34’s complex provided a dual capability. Test engineers could proceed in the traditional manner - initiating commands from switchboards and checking the results on meters, strip recorders, and cathode ray tube displays. In this mode, testing would remain essentially manual, with the computer complex serving as an expensive data link. Or the launch team could convert the manual test procedures into computer programs for interpretation and execution by the 110As. During the early IB missions, manual procedures predominated. Besides the inevitable resistance to change, systems engineers had trouble converting their test procedures into machine language. Development of a special computer test language alleviated the latter problem, while Gruene’s leadership prevailed over personal inertia. By the end of Apollo, most launch vehicle tests were fully automated programs.17

Saturn ground computers employed two types of programs: operating system and test. As the name implied, the operating system program was the computer’s basic software. It operated continuously, seeking alternate paths when a failure was detected. Manual testing of the Saturn vehicle was accomplished through this program. Test and monitor programs provided the means to automate the checkout process; they were the system engineer’s primary software tool. Test programs could be prepared in machine language, using the full capability of the computer’s logic, or in Atoll (Acceptance Test Or Launch Language), a form of engineer’s shorthand. Once prepared, the test programs performed a number of important tasks: sequencing required events during a test, evaluating system responses, monitoring, displaying anomalies, and tying together a series of programs. A test program could preselect the operation sequence, change the limits of tested values, and intervene during the operation. On the later Apollo missions, test programs accomplished much of the routine checkout. Engineers would initiate programs through console keyboards and react when problems arose. Many of the tests would start on their own - the launch team having programmed the computer to call up test programs at a certain time in the countdown or when another test was successfully completed. While test programs were the key to automated checkout, maintenance and post-test processing programs played an important role. Maintenance programs tested the interfaces between RCA 110As and related equipment. With the post-test processing programs, engineers converted raw data into usable printouts.18

Changing test procedures into computer programs may well have been the highest hurdle in Saturn automation. Early Saturn I automation rested largely with IBM, Huntsville’s contractor for computer software. From KSC requirements, IBM programmers and system analysts prepared machine-language programs. The Automation Office provided the coordination between Saturn test engineers and IBM computer experts. Unfortunately, it was not simply a matter of converting a few lines from English into a computer program. KSC’s manual procedures did not spell out every detail; many contingency actions depended upon an engineer’s intimate knowledge of a system. Inevitably, misunderstandings arose. Some KSC systems engineers viewed the process as a one-way street: IBM programmers were gaining a knowledge of Saturn hardware while KSC engineers learned little about automation. Furthermore, as Saturn automation grew, requirements for IBM support increased. Clearly, KSC needed some way to simplify the conversion from test procedure to computer program - a route that would bypass machine language.19

The solution was Atoll, a computer language under development at Huntsville’s Quality Laboratory. By 1965 the Astrionics Laboratory and IBM were incorporating Atoll into IB plans. KSC’s automation team helped define launch site requirements for the new software system. The AS-201 mission in February 1966 used only a half-dozen Atoll procedures, but subsequent launches showed increasing numbers as KSC engineers converted from manual procedures to computer programs. There were 21 Atoll programs on AS- 501, 43 on AS-506 (Apollo 11), and 105 by the Apollo 14 launch in early 1971. Atoll proved particularly valuable for Saturn systems that changed from one mission to the next. While modifications to machine language procedures required approval from Marshall, KSC’s Automation Office controlled Atoll. Changes could therefore be approved quickly at the launch center. More importantly, Atoll involved the test engineers directly and its use was instrumental in their acceptance of computerized checkout.20

Automating Telemetry Operations

The move from the Redstone to the Saturn era brought a pressing need to automate data reduction.* The Saturn I’s telemetry, the primary source of postflight test data, represented a three-fold increase over previous rockets and the Saturn V would be an even greater jump [see table 1, chapter 2-1]. The increase posed a problem of space. Continued reliance on analog strip charts would have forced Saturn V engineers to review thousands of feet of chart after each launch. Time was a second consideration. In September 1961 Fridtjof Speer, chairman of Marshall’s Saturn System Evaluation Working Group, expressed concern about possible delays in the delivery of postflight data. Although LOD had agreed to submit telemetry data within 12 hours, Speer wanted backup blockhouse records (strip charts and event recorders) within 24 hours. He asked Debus to “exploit every possible course of action to satisfy this requirement.”21

A month after Speer’s letter, Dr. Rudolf Bruns’s data reduction team pioneered the use of computers in Saturn I launch operations. Digital computers offered two advantages. During a launch the computer could record incoming telemetry data on a magnetic tape and subsequently process compact printouts in a relatively short time. The computer, supported by peripheral equipment such as cathode ray tube consoles, could display critical information in real time.22

The Flight Instrumentation Planning and Analysis Group set up a Burroughs 205 computer alongside the telemetry station in hangar D prior to the SA-1 launch. A tarpaper shack housed the computer - a far cry from facilities the team would enjoy in a few years. Despite its primitive surroundings, the 205 provided guidance data and some measurement reduction in real time. A General Electric computer replaced the Burroughs machine on SA-3; the GE computer’s solid state circuitry and core memory provided a faster sampling of Saturn telemetry.23

During the last hours of the SA-3 countdown, the launch team periodically tested telemetry transmitters and the data reduction computer. The team “dumped” the telemetry data from the GE computer at several predetermined times, taking a quick look at the printout to see if the measurements were within calibration. At T - 30 minutes the computer began processing data from the Saturn’s ten commutators. The GE 225 took approximately ten seconds to complete one sample of the Saturn’s ten telemetry links; most of that time was taken up by the telemetry station switching device, switching from one commutator to the next. As the 225 printer could not match the computer’s calculating speed, the real-time printout listed only the out-of-tolerance values. The GE 225 processed telemetry data until the signal faded a few minutes after launch. During postlaunch activities, the telemetry station rewound its analog tapes and played them back for the computer. The digital magnetic tape data, produced by the playback, were then converted in another computer process to specific engineering units (e.g., degrees Celsius), which were displayed on a printout or a Stromberg-Carlson 4020 plotter.24

Efficient data reduction depended on parallel advances in the telemetry station’s digitizing system, the equipment that converted the Saturn’s analog telemetry into a digital message for the GE computer. The earliest digitizer employed one analog-to-digital converter and one synchronizer. Each of the ten commutators on SA-3 fed its measurements (27 each for the low-speed commutators and 216 for the high-speed commutator) into a subcarrier oscillator. The oscillators sent the signals out over the launch vehicle’s six telemetry transmitters as a pulse-amplitude-modulated wave. The receiver in the ground telemetry station removed the subcarrier and directed it to one of three discriminators where it was demodulated. The analog-to-digital converter changed each signal voltage from a magnitude to a pulse. A 0.6-second delay in synchronizing the converter’s switch from one commutator to another limited the digitizer’s output to the GE 225. Due to the switching delay, the computer received approximately one measurement of every 120 that came from the Saturn’s high-speed commutator.25

Work on a faster digitizing system began in mid-1962. An analog-to-digital converter was added for each vehicle commutator; a parallel programmer-addresser generated a 12-bit address word to identify the data. A digital scanner, essentially an electronic 16-position switch, scanned the outputs of the data channels (convertor and addressor), transferring new data to a core memory. The memory stored each measurement according to its digital address and provided the computer with random access to any data. Although the telemetry team experienced problems interfacing the scanner and core memory, the specifications were ready by November.26

The telemetry scanning and digitizing system was added to a GE 235 computer for the block II series of the Saturn I program. The GE computer, given immediate access to all data through the digital scanner and core memory, recorded data in real time on a magnetic tape. Since the 235’s printing lagged behind its computations, real-time display was still limited. The 235’s other functions included: separating data by program (S-I, S-IV, instrumentation unit, and spacecraft) and measurement, converting measurements to engineering units, arranging data for use on the 4020 plotter, and comparing engineering units versus time on printouts and 4020 plots (microfilm or hard-copy graphs). Within two hours of the SA-7 launch, all the 4,020 plots had been processed. The total data reduction program was completed eight hours later.27

Measuring techniques needed to be improved to keep up with the advances in digitizing and telemetry reduction. Automation of measurements during checkout of the Saturn vehicle was begun in March 1962, using an IBM card system. Punched cards, placed in a card reader, selected the appropriate channel (and relay if calibration was necessary). The card reader compared the signal returning from the launch vehicle’s measuring device with data prepunched on the card and gave a “go, no-go” evaluation.** Following experimentation during a checkout on LC-34, the system was installed in LC-37’s measuring station for the block II launches.28

As the automation plans gained momentum, Debus expressed concern about their impact on LOC’s relations with the Air Force. On 21 February 1962 Debus penned a brief note to Gruene’s weekly report:

Hans: One day we have to start an analysis of what this entire automatic checkout with computers will mean in our countdown-and-test interfaces with the range! For instance: timing, on-off commanded to the Range, TM [telemetry] receiving . . . and a host of other interactions. . . . Does it still make sense to plan a “joint” TM station. . . ?29

After Debus and General Davis discussed some of LOC’s scheduling problems in May, Air Force and NASA officials held a series of meetings that summer, some as the Joint Instrumentation Planning Group, others in informal sessions. Their work led to the development of a new telemetry station and the central instrumentation facility.30

  1. Data reduction means the transformation of observed values into useful, ordered, or simplified information. With telemetry it could involve transferring an analog electrical signal onto a brush recording or eliminating unnecessary portions of a message (e.g., the address), restructuring the data, and directing it to various users.
  2. A go, no-go indication told the operator whether a device was functioning properly, without indicating how far out of tolerance it might be or what was wrong.

Automatic Checkout for the Spacecraft

Automated checkout of the Apollo spacecraft had its origins at Cape Canaveral in 1961. Preflight Operations Division engineers, members of the Space Task Group, realized that Mercury launch methods would not satisfy Apollo requirements. The Mercury preflight tests resembled an aircraft checkout. One test team worked from a command post near the spacecraft while a second group monitored the test results at a remote station. During the checkout hundreds of wires ran through the open hatch into the cockpit, leaving barely enough room for an astronaut or a test engineer. There were other limitations. During the prelaunch operations, the spacecraft would likely move several times, and each move required disconnecting and reconnecting the various test lines. As the checkout grew more complicated, the test conductor found it increasingly difficult to coordinate activities at the spacecraft and monitoring station. Less than 100 telemetered measurements in Mercury had occupied the Instrumentation Branch. The 2,000 measurements projected by Apollo feasibility studies made some form of automated checkout inevitable.31

Following a September 1961 briefing on Apollo, G. Merritt Preston, Preflight Operations Division’s chief, asked his staff to consider the proposed spacecraft’s impact on launch operations. Jacob Moser and his Flight and Ground Instrumentation chiefs, Walter Parsons and Harold Johnson, responded with an automation proposal, and Preston gave the project a green light. Mercury operations limited progress during the next two months, but with further urging from Preston, the instrumentation team formalized a presentation in December. Two young engineers, Thomas Walton and Gary Woods, joined in this early conceptual work. For their efforts the five subsequently won a patent on the checkout system. The group’s pre-Christmas briefing favorably impressed the staff. A Marshall delegation displayed less enthusiasm but failed to halt the project.32

The automation team began the new year with a search for available equipment. Since money was scarce, only off-the-shelf hardware could be used. Walton and Woods scoured American factories, finding all the necessary components except a digital command system. At the same time Preston secured the support of Robert Gilruth and Walter Williams, the Director and Associate Director for NASA’s Manned Spaceflight Center. In February the team conducted a series of formal briefings for NASA’s manned spaceflight organizations and for supporting contractors. The road show, complete with a projector and more than 500 slides, drew a mixed response. Headquarters officials questioned some of the team’s technical assumptions (e.g., James Sloan, OMSF’s Deputy Director for Integration and Checkout, doubted that the software planned for the system could be perfected). The principal user, North American, perhaps hoping to develop a checkout system itself, was particularly critical of the concept. Despite the numerous objections, acceptance checkout equipment (ACE)* was approved by mid-1962.33

While gaining support within NASA was, perhaps, the most difficult hurdle, the design also involved some challenges. The spacecraft had not yet been clearly defined when the group began work on a report in February 1962. Woods concentrated on the system’s uplink. As the name implies, the uplink carried commands from operator consoles to the spacecraft via coaxial cable or radio. Woods demonstrated the feasibility of his uplink in June, using 32 kilometers of cable stretched from a Patrick Air Force Base command post to the Cape. Meanwhile Walton pursued the problems of the downlink, the portion of the checkout system that brought encoded signals from the spacecraft, through a decommutator and computer, to display devices. Johnson focused on another part of the downlink, the analog display recorders.34

In July the acceptance checkout equipment team began procuring equipment for an experimental station at the Cape. Gemini officials helped fund the laboratory in hopes that the system might benefit their program. The Instrumentation Branch activated the station in September; its original equipment consisted of a small computer, an alphanumeric display device, a decommutation system, and the manual uplink prototype. A downlink prototype was put in operation the following month. By April 1963 the team was working two digital computers in a non-synchronized mode, exchanging data through a shared memory base. Gordon Cooper’s 22 revolutions around the world in May 1963 marked another milestone for the station. The experimental equipment provided real-time support of preflight checkout and inflight operations for the last Mercury mission. The station’s computers displayed Faith 7’s telemetry data on screens and high-speed line printers. The laboratory was fast becoming one of the tourist attractions at Cape Canaveral; during their visits to the Cape, new astronauts spent a half-day in the station.35

The General Electric Company entered the ACE story in November 1962. GE’s Apollo roles, as delineated by NASA management, included the development of “overall system checkout equipment” [see chapter 9-1]. Since ACE would test North American’s command and service modules and Grumman’s lunar module, the checkout system fell within GE’s area of responsibility. At first GE provided engineering support. Within three months Leroy Foster had 20 engineers working on equipment specifications. The decision at NASA Headquarters to have GE produce the Apollo checkout stations (as a modification to its existing contract) touched off ten months of proposals and counterproposals. The main dispute between GE and Cape officials centered on the issue of government-furnished equipment. The Preflight Operations Division intended to provide GE most of the components, buying parts already developed by other companies. GE, understandably, thought it could improve on some of the equipment. At a stormy July session in Daytona, Jack Records, GE’s number two man at the Apollo plant, and Dr. Lyndell Saline questioned the suitability of Control Data Corporation’s 160G computer. When Preston asked for proof of the computer’s inadequacy, however, the GE executives withdrew their charge.36

Negotiations with General Electric were complicated by officials at NASA Headquarters; Joseph Shea, OMSF’s Deputy Director for Systems, supported GE. In September 1963, he called the ACE team to Washington for a showdown on the spacecraft checkout. Shea and his Bellcomm** advisors attacked ACE on several grounds, including insufficient memory and interrupt capability. Cape officials refuted the criticisms point by point. Before the end of the day Shea had given up his opposition to ACE.37

After settling the issue of government-furnished equipment, GE and the Florida Operations group (the new name for Houston’s launch team at the Cape) moved swiftly to meet the September 1964 deadline for the first operational ACE station. At the Cape, Douglas Black’s team conducted a series of critical interface tests at the experimental station in the first half of 1964. By June the first computer programs had been verified. GE shipped components for the first station to Downey, California, in July. Within 60 days North American was using the station to check out Apollo 009, the spacecraft that would fly on AS-201. GE installed 13 more ACE stations: 2 at Downey; 3 at Grumman’s Bethpage, New York, plant; 2 in Houston; and 6 at the Cape. KSC’s first station became operational in March 1965.38

  1. ACE was initially SPACE, Spacecraft Prelaunch Automatic Checkout Equipment. Cape officials changed the title to Prelaunch Automatic Checkout Equipment for Spacecraft, PACE-S/C only to find that PACE was already a legal name. They then dropped the Prelaunch and changed the Automatic to Acceptance.
  2. Bellcomm, Inc., was a subsidiary corporation of AT&T, organized to assist OMSF’s Systems Office in the overall integration of Apollo. The work resembled that being done by GE, but was at a higher level and on a much smaller scale.

Spacecraft Checkout

ACE control room
An operation under way in the automatic checkout equipment (ACE) control room, February 1967.

ACE’s first major test at KSC came with the checkout of Apollo 009 (AS-201 mission) in late 1965. The spacecraft team directed the checkout from a control room in the operations and checkout building. Engineers from Spacecraft Operations and North American, working in pairs, tested the nine functional systems: communications, instrumentation, service propulsion and reaction control, stabilization and control, guidance and navigation, power and sequence, fuel cell and cryogenics, aeromedical and astronaut communications, and environmental control. Commands were initiated at the test consoles, e.g., an engineer might test the freon level in the command module’s environmental control system. His signal went to the command computer for conversion to a binary instruction. The digital message traveled a complicated electrical path to the spacecraft, where it triggered a sensor in the command module. The sensor noted the condition of the freon and transmitted an appropriate response. Data acquisition equipment routed the signal back to a display computer, which processed the message for presentation on the same test console whence the command had come seconds earlier. Command and display computers and much of the data acquisition and recording equipment were located in an ACE computer room.39

ACE room
Automatic checkout equipment room, February 1967.

Three different groups of sensors obtained data concerning the Apollo spacecraft: ground service equipment, carry-on equipment that was removed prior to flight, and sensors built into the spacecraft. Coaxial cable and radio connected the various sensors to the control rooms in the operations and checkout building. There, data traveled through one of three different paths. The most important, from the standpoint of real-time display, was the display computer. Its functions included: comparing machine words to determine whether data fell within predetermined limits, converting data into engineering units (such as heat rise in degrees per second), and generating signals that would produce alphanumeric displays on consoles. The display system was impressive but not foolproof. An engineer recalls that on its first day of operation, the console welcomed them: “GOOD MORING.” ACE had failed its first spelling test.40

During lunar missions, four control rooms would be used for spacecraft checkout: primary and backup rooms for the command and service modules and another pair for the lunar module. Each room had 20 master consoles and additional slave consoles. The latter displayed the same data shown on a master, but did not provide the means to select information. Nine TV monitors carried pictures from portable cameras located around the spacecraft. The overhead monitors were part of an operational TV network that carried spacecraft and launch vehicle pictures to the launch control center and central instrumentation facility, as well as the operations and checkout building. Although the equipment had a similar appearance, configurations differed, depending on the requirements of particular systems. During checkout, between 40 and 50 men occupied each of the primary control rooms. In the backup rooms, the consoles were kept in operation but usually were not manned. Each control room was supported by a computer room with its uplink and downlink equipment.41

ENDNOTES

  1. Sidney Sternberg, “Automated Checkout Equipment - The Apollo Hippocrates,” in Man on the Moon, ed. Eugene Rabinowitch and Richard Lewis (New York: Basic Books. 1969), pp. 196-97.X
  2. W. Haeussermann, Dir., Guidance and Control Div., MSFC, memo for record, “Meeting on Saturn Checkout Equipment,” 22 July 1960; Paul interview.X
  3. Debus to Dieter Grau, “Automatic Checkout Committee,” 2 Sept. 1960; Richard interview, 12 Dec. 1973. See B. J. Funderburk, Automation in Saturn I First Stage Checkout (NASA TN D-4328, Jan. 1968), for story of the Packard Bell 250 and MSFC’s early automation efforts.X
  4. Ludie Richard and Charles O. Brooks, The Saturn Systems Automation Plan, MSFC. 15 Sept. 1961, sec. II.X
  5. Ibid., sec. VII.X
  6. "Brief Chronological History of the Saturn V Breadboard,” attached to MSFC Automation Plan, 8 May 1962; Burns interview; Greenfield interview.X
  7. Jafferis interviews, 19 Dec. 1973, 22 Jan. 1974; Greenfield interview; Whiteside interview. 4 Jan. 1974.X
  8. Jafferis interview, 19 Dec. 1973; “Description for Use of Saturn Ground Computer on SA-5,” draft copy in Jafferis’s private papers; Spaceport News, 21 May 1964, p. 2.X
  9. B. E. Duran, “Saturn I/IB Launch Vehicle Operational Status and Experience,” read at Aeronautic and Space Engineering and Manufacturing Meeting, New York, 7-11 Oct. 1968, Society of Automotive Engineers reprint 680739; KSC, “Utilization of Saturn/Apollo Control and Checkout System for Prelaunch Checkout and Launch Operations,” GP-663, 25 Mar. 1969.X
  10. W. O. Frost and D. E. Norvell, “Telemetry System Design for Saturn Vehicles,” Proceedings, 1966 International Telemetering Conference, Los Angeles, 18-20 Oct. 1966. p. 70. See also E. A. Robin, “Development and Utilization of Computer Test Programs for Checkout of Space Vehicles,” p. 297; Canaveral Council of Technical Societies, Proceedings of the Second Space Congress, Cocoa Beach, FL, 5- 7 Apr. 1965, pp. 617, 634 D. M. Schmidt, “Automatic Checkout Systems for Stages of the Saturn V Manned Space Vehicle,” International Convention Record of Electrical and Electronics Engineers 13 (pt. 4, Mar. 1965), p. 87.X
  11. Canaveral Council of Technical Societies, Proceedings of the Second Space Congress. p. 656; William G. Bodie, “Techniques of Implementing Launch Automation Programs. Saturn IB Space Vehicle System,” Practical Techniques and Applications, 4: 740. See also Apollo/Saturn IB Launch Operations Plan AS-203, KSC document K-IB-021.3, p. 6-8.X
  12. Duran, “Saturn 1/IB Launch Vehicle Operational Status"; KSC, “Utilization of Saturn Apollo Control and Checkout."X
  13. Richard Dutton and William Jafferis, “Utilization of Saturn/Apollo Control and Checkout System for Prelaunch Checkout and Launch Operations,” paper read at New York Univ., Project SETE, 24-28 July 1967, pp. 3-34 through 3-43; Medlock interview; Thompson interview.X
  14. F. Brooks Moore and William Jafferis, “Apollo/Saturn Prelaunch Checkout Display Systems,” read at IEEE Conference on Displays, Univ. of Loughborough, England, 7-10 Sept. 1971, pp. 7-9.X
  15. Ibid., pp. 9, 15-16.X
  16. Ibid., p. 14.X
  17. Richard Jenke to Benson, 17 Jan. 1975; Richard Smith interview; Medlock interview; Thompson interview.X
  18. Dutton and Jafferis, “Utilization of Saturn/Apollo Control and Checkout System,” pp. 3-44 through 3- 48; Jenke to Benson, 17 Jan. 1975.X
  19. Jenke to Benson, 17 Jan. 1975; Medlock and Thompson interviews.X
  20. Jenke to Benson, 17 Jan. 1975; Medlock and Thompson interviews.X
  21. Fridtjof Speer, Chairman, Saturn System Evaluation Working Gp., MSFC, to LOD Dir., “Justification for Early Delivery of the Saturn Blockhouse Records and Sequence Records,” 19 Sept. 1961, Debus reading file.X
  22. R. W. Bivans, G. D. Matthews, and F. T. Innes, “A Scanning and Digitizing System for Multiple Asynchronous Telemetry Data Sources,” read at National Telemetry Conference, Los Angeles, June 1964, p. 1, G. D. Matthews’s private papers.X
  23. Bruns interview, 3 Jan. 1974; Bobby Griffin and G. D. Matthews, The Real-Time Telemetry Data Processing Effort at the Launch Operations Center, MTP-LVO-63-2, MSFC,X
  24. Griffin and Matthews, Real-Time Telemetry Data, pp. 9-11; Bruns interview, 3 Jan. 1974; Corbett, Hughes, and Jelen interviews.X
  25. Griffin and Matthews, Real-Time Telemetry Data, pp. 3-6; George Matthews interview.X
  26. Griffin and Matthews, Real-Time Telemetry Data, pp. 14-17; Bivans, Matthews, and Innes, “Scanning and Digitizing"; George Matthews interview; LOC Weekly Notes, Sendler to Debus, 23 Aug., 15 Nov. 1962.X
  27. KSC Computation Br., “Scientific Computation Support of Saturn/Apollo Vehicle, SA-7,” TR-103-2, 3 Dec. 1964.X
  28. Joralan interview, 3 Jan. 1974; LOC Weekly Notes, Sendler to Debus, 22 Mar. 1962.X
  29. LOC Weekly Notes, Gruene to Debus, 21 Feb. 1962.X
  30. Raymond Clark, Asst. LOC Dir., to Col. Max Carey, “Request for Additional Data on NASA Telemetry Requirements,” 6 Oct. 1962; Debus to Davis, “The AFMTC Launch Area Telemetry System Plan, 28 September 1962,” 18 Oct. 1962.X
  31. Telephone directory, Project Mercury Field Ops., STG, Cape Canaveral, FL, Sept. 1961; “Patent Application on ACE, NASA Case No. 8012,” encl. to letter, James O. Harrel to Harold G. Johnson, 20 Jan. 1967, Johnson’s private papers; Walton interview, 17 Dec. 1970. The Cape launch team first appeared as Preflight Operations Division on a Sept. 1962 MSC organization chart. Earlier it was called Mercury Field Operations or MSC’s Atlantic Missile Range Operations.X
  32. Parsons interview; Preston interview, 22 Jan. 1974.X
  33. Parsons interview; Walton interview, 17 Dec. 1970; Preston interview, 22 Jan. 1974; W. E. Parsons, Head, Flight Instrumentation Sec., to C. W . Frick, Head, Apollo Project Off., MSC, “Implementation Plan for Apollo SPACE System,” 26 June 1962, Johnson’s private papers; “PACE-S/C History,” compiled by Harold Johnson ca. 1963, Johnson’s private papers.X
  34. Parsons interview; Harold Johnson interview; “PACE-S/C History"; Walton interview, 17 Dec. 1970.X
  35. Tom S. Walton, MSC Florida Ops., Experimental Station Implementation and Planning, 18 Dec. 1964; Walton interview, 17 Dec. 1970.X
  36. "PACE-S/C History"; Parsons interview; Walton interview, 23 Jan. 1974.X
  37. Parsons interview; Norwalk interview; Walton interview, 23 Jan. 1974.X
  38. Parsons interview.X
  39. Page interview; Spaceport News, 6 Jan. 1966; Apollo Support Dept., General Electric Co., ACE-S/C, Acceptance Checkout Equipment, Spacecraft, Daytona Beach. FL. undated.X
  40. Apollo Support Dept., General Electric Co., ACE-S/C; James O. Hassell to Harold G. Johnson, “Patent Application on ACE,” 20 Jan. 1967, with encl. 1, “Patent Application on ACE, NASA Case No. 8012,” Johnson’s private papers; Harold Johnson interview.X
  41. Moore and Jafferis, “Apollo/Saturn Prelaunch Checkout,” pp. 4-7.X