Astronauts enhance Space Station science operations
James Oberg / SPACEFLIGHT magazine (London), August 2004 / pp. 312-314

NASA materials scientist Richard Grugel was operating his delicate crystallization experiment one afternoon when he noticed a passerby tug on a strap on the outside of his apparatus. The device jerked sharply – he saw the forces on it – and a small bubble he had created on his sample in the furnace suddenly broke loose. Actually, he had been trying to get the bubble away from the main sample, so he was pleased by the accident.

After all, he had no means himself of tapping the box manually, or shaking it in any other way. He was on Earth, and the apparatus he was controlling was on the International Space Station thousands of kilometers away.

But his experience that day highlighted the widely unrecognized higher levels of ‘situational awareness’ for ground controllers such as Grugel who operate equipment in the laboratories aboard the space station from their ground-based telescience facilities, from their office computers, and even from their wireless laptops at home or on travel.

“The move to telescience, that’s one of the most wonderful things we can do on the station,” stressed Dr. Merri Sanchez, a NASA official in Houston who has served as ‘Increment Manager’ for several of the recent expeditions. “We have payloads continuously operating,” she told me, whether the crew is present or not, and whether the station is in radio contact with Earth or not.

Scientific research is indeed being conducted aboard the International Space Station, even during the current ‘caretaker’ period when a reduced crew keeps the facility going until space shuttle flights resume sometime late in 2004 or early 2005.

The station is the most “science-friendly” station in history.

Yet even before the ‘Columbia’ disaster on February 1, 2003, there had been serious questions about the station’s very low efficiency of science operations. Contrasted against what the science community had been promised in terms of crew time on orbit, the station was running at about 10 to 15% of the promised efficiency – and that, at much higher cost than originally planned for 100%.

Since the station was permanently occupied late in 2000, it’s gone through several phases. First was a year of rapid-fire assembly of modules, through the end of 2001. Then there was about a year of shake-down during which the station’s equipment was deployed, brought on line, and utilized. Last year, 2003, was supposed to be a second ‘big push’ of assembly missions to achieve what NASA came to call the ‘Core Complete’ stage. But even then it was so short of what had once been called ‘Assembly Complete’ (which included key laboratory modules from Europe and Japan, and a habitation module to double the crew size) that cynics dubbed the partial ‘Core Complete’ stage as ‘Half-Assembly Complete”.

And now, as NASA struggles with returning its three remaining space shuttle orbiters – Atlantis, Discovery, and Endeavour – to flight, prospects are fading of the space station ever fulfilling the bold promises once made for it. And more’s the pity, since when the station does perform real research, it does it very, very well – but just nowhere near enough, critics say, to justify the cost.

For example, Ricky Cissom, manager of the Payload Operations Center at NASA-MSFC, helped me understand the impressive technological capabilities of the communications links. “The technology is leaps and bounds ahead of early Shuttle and Apollo,” he said. Downlink is 50 megabits/sec via a Ku-band antenna, and this includes data and up to four television channels. New ground processing software is supposed to triple that rate by the end of 2003.

As a result, experiments such as Grugel’s can be operated successfully and genuine research can be conducted. Grugel’s project is called ‘Pore Formation and Mobility Investigation’ (or ‘PFMI’), and in 2002 it became one of the first two materials science experiments performed aboard the ISS. With different samples and different procedures, it has continued through subsequent expeditions.

On Earth, bubbles that form during the fabrication of metal or crystalline materials – that’s what ‘porosity’ refers to – are defects that diminish the product’s strength and usefulness. The PFMI experiments use a transparent modeling material, succinonitrile (either alone or mixed with water) to observe how bubbles form, move, and interact during melting and directional resolidification. Results so far have revealed features of the process that promise to improve industrial production methods.

“To set up the experiment, the glovebox controllers do the videotaping, and we control the experimental hardware,” Grugel told me . “We can change the processing parameters in real time – for instance, sample temperature, or the rate we’re translating the sample. We have visuals – we have two cameras in the experiment hardware, we are watching our sample anywhere from 1X to 40X magnification, and we can move the camera up and down the samples.”

This is all done under remote control, he explained: “The crew is usually off doing something else.” But the ground team pays full attention during the entire process: “We usually run ten to twelve hours – then the sample will be returned. The more important part is the video.”

Operating out of a special facility at NASA’s Marshall Spacecraft Center in Huntsville, Alabama, Grugel says that it seems “the device could be in the building next door.” There is only a slight delay: “We send a command a see a response in about five seconds. Distance and delay isn’t an impact, the biggest impact is the breaks in the signal.” POC Manager Cissom corroborates Grugel’s round trip signal estimate: “The best we do round trip was three and a half to four seconds, the worst I’ve ever seen was in the seven second range.”

Grugel’s main complaint is in how often – and for how long – the real-time link is interrupted. Originally (and ultimately) the comm link is supposed to be practically continuous, but so far it’s far from that goal: “We didn’t get 100% viewing – we get twenty or thirty minutes an hour in different sized segments,” Grugel explained.

According to Cissom, “We average 30-35% coverage over 24 hours, sometimes almost a continuous rev of good coverage, sometimes several revs with only sporadic coverage.”

“It’s a function of blockage of vehicle structure – when the trusses are completely built out, and the antenna is moved, we’ll approach 80% coverage – the theoretical maximum is 92%.”

The antenna is atop the Z1 truss right on the roof of the Destiny lab, and temporarily there is a set of arrays and radiators directly above it, that will be moved out to the end of a new array to be added once shuttle flights resume.

Even in the current configuration, ground operators have accumulated enough experience to operate the antenna system with improved efficiency. Cissom described how his team
“improved [coverage] a bit with better software to analyze antenna pointing and signal line-of-sight versus structure.” At first, he went on, “we were afraid we could damage hardware, so we would turn off the antenna in large angular areas.” They were also getting some reflectivity off trusses, so they added software to recognize this. “And we developed a pretty sophisticated model to anticipate antenna line of sight,” he said.

Merri Sanchez, who works directly with the main Mission Control Center in Houston, gave another example of the ‘learning curve’ to get the most efficient comm out of the system. “We have a small gimbel heater failed in the Ku-band antenna, “she explained. “Depending on attitude, if it gets too cold we have to shut down.”

“In earlier days it was a significant impact,” she continued. “Then we operationally learned to predict it, and we’re getting more experience. We have certified additional attitudes, and now I don’t remember the last time this was a constraint.” As a result, NASA has no plans to repair or replace the heater – they’ve learned to live with its failure. “We’re getting confident with our thermal models,” Sanchez concluded. “We’re a lot more comfortable with our operational flexibility now.”

As a result of such experience, while the outages are unavoidable, they are not unpredictable. “They have these very well scheduled, long in advance,” Grugal pointed out. “Five minutes on, nothing for ten, one minute on, then fifteen off, then twenty on… There’s no obvious pattern. It depends on the attitude of the station. In [solar orientation] you can have four hours at a stretch, then nothing for five hours. In [horizon orientation], it’s more sporadic.”

The blockage affects only continuous monitoring of these experiments. “Core systems telemetry and all vehicle commanding are sent at a low data rate over S-band,” Cissom explained, “and it has coverage in the 70’s to 80’s percentage.” And no data is lost during these blockage periods, Grugel added: “The video is continuous on the station, you can dump that later.”

Using the current communications links – and despite the interruptions they are a vast improvement over all previous space stations – payload operators have extended their earthside networks to provide links between actual scientific investigators and their hardware in space. “We upgraded our control center to allow payload developers to access and command from their offices, from standard laptops, “Cissom explained. “It’s a fully Internet-based system with password protection and encryption” and has been operating since early 2001.

“We’ve had 46-47 remote sites – at any one time, 20 to 30 are involved in experiments,” he continued. It is to be upgraded to handle 200 remote sites soon. As an example, on one experiment rack (called ‘Express-1’) there are eight subracks, and each one could be operated from any of the sites. Another rack, the ‘Human Resource Facility’, is controlled from NASA-JSC in Houston, which distributes live data to 20-30 principal investigators at remote sites.

All of these users are not jamming the bandwidth simultaneously. “We develop the ‘command plan’ to spread the commands out,” Cissom explained. “The most sites in operation simultaneously has been three or four, since we try to schedule around conflicts.”

The resulting operational mode for the station’s science experiments is not for the crew to continuously monitor and control and experimental processes. Instead, they assist in start-up and adjustments and close-out, as well as troubleshooting. “We don’t have the luxury of calling them up every twenty minutes for an adjustment,” Grugal admitted. “But a few times we had to ask them to check out a strange reading.”

Of the approximately hundred and fifty scheduled man-hours per week, science gets only a small fraction. The hours include, in Sanchez’s words, “all preparation, procedures review, gathering tools, all actual research activity like changing out samples, all cleanup. We try to guarantee twenty hours a week [but] we’ve averaged just under ten.”

Even though the crew size was reduced to two men in April 2003, the cancellation of previously scheduled shuttle-supported assembly and spacewalking tasks had freed up enough hours to compensate. “There are as many hours for payload as in the original increment plan,” she told me. “We have the time – but not the payloads,” or the ability to restock experimental materials or return finished products to Earth.

How could science operations be improved? Grugal’s wish-list mirrored comments from other investigators: “Full time monitoring, and more than just two views,” he told me, “and the ability to focus sharper on something a little more interesting.”

That final wish for pure video at very high resolution is also in work. According to Cissom, ISS comm will be able to do a 43 to 47 megabit/sec video stream (cf NTSC commercial TV of 1-2 megabits/sec). They already plan to do a HDTV transmission experiment with Japanese equipment when shuttle flights resume. “It’s a wonderful camera,” adds Sanchez. “And it’ll stay there – provides extra resolution of imagery, and will improve PI [principal investigator] insight into experiments. “

As a result of the teleoperations capability, experienced operators who have extended their senses and their controllers onto the station often almost ‘feel’ they are there. “We see ourselves as the fourth and fifth crewmen on orbit,” Cissom told me. “And we’re running 24 hours a day with remote commanders.” That’s already better presence than the crewmembers on board.

[End of as-published article – this later section was deleted].

Simply fiddling with the apparatus aboard the station shouldn’t be sneered at, either. Despite the long-range science telepresence, often what a space experiment needs is less knowledge and more hands-on dynamics.

A good example of the importance of an on-orbit crew’s ability to repair equipment is the research facility commonly called the ‘glovebox’. Designed to isolate potentially hazardous activities from the cabin atmosphere (a requirement for a human space facility), the unit has a clear plastic chamber with neoprene gloves to allow manual manipulation of the isolated apparatus and samples. It has hosted numerous different materials processing experiments, amounting to about a third of all scientific research performed on ISS.

On November 20, 2002, within days of the end of her space tour, Expedition Five ‘Science Officer’ Peggy Whitson was changing a tape on the unit’s video recorder when she heard a ‘click’ – a circuit breaker cutting power. A shuttle mission was about to deliver a new crew, and during her handover to Don Pettit, both scientists ran connectivity tests. They isolated the problem to one of two electronic panels for distributing electricity. These panels were returned to Earth on the shuttle that delivered Pettit and his teammates, and were subsequently returned to the ISS aboard an unmanned Russian spacecraft early in February. After the unit was restarted, it exhibited signs similar to those that preceded previous failure. Pettit spend many hours troubleshooting the circuits to discover the cause of the original circuitbreaker trip, and by the end of February had it safely operating.

Merri Sanchez, the NASA ‘Increment Manager’ for station expeditions, stressed the significance of this success in an interview with me. The glovebox repair, she said, was “a prime example of being able to have the crew observe what’s going on, for on-orbit analysis to give the ground data to troubleshoot. The crew can see a wire broken, can see condensate here, then gives the information to the repairmen to take ground repair procedures.” This was part of a larger pattern: “Expedition Six was amazing,” she continued. “They did on-orbit repairs we never would have tried. We would have brought the hardware home, but we were forced to try these repairs.”

And the glovebox was the key to much of the science capability of the station. A genuine example of ISS science operations at their best has been the Zeolite Crystal Growth experiment. Early in January 2003, Expedition Six Commander Ken Bowersox unloaded zeolite samples that had been completed two days earlier in a test furnace. He then reconfigured the furnace for another round of tests. First he had to manually twirl each of the 19 new sample tubes to reduce the number of bubbles in them, and then he installed the samples in the ZCG furnace to begin a scheduled 15-day processing run.

On the ground, the Center for Advanced Microgravity Materials Processing at Northeastern University in Boston sent commands to initiate mixing of the samples. One sample appeared to jam, and Bowersox was called to help. He used a hand drill to mix the sample and begin processing.

“The autoclave malfunctioned due to the particular sand-like nature of the mixture,” said Vic Cooley, Expedition Six Lead Increment Scientist. “Although this experiment is controlled from the ground, Ken Bowersox’s ability to intervene, assess the problem and take corrective action enabled the science team to begin processing all of their samples. It’s also a good example of how humans in space and humans on the ground can work together to achieve more than they could separately.”