What a great story. Quite heartening to find out about Zotterelli's accomplishments, after all, he started working on Voyager when he was 42. Goes to show, "senior" engineers are very capable of making noteworthy contributions. (In reflection of a recent HN thread about ageism in software industry.)
The article also points out the extreme importance of documentation, because we never know what systems will still be running, or programs still in use 40 years in the future. It's truly terribly wrong to "take secrets to the grave", knowledge should be shared, we all have a stake in it.
I fervently wish Voyager's secrets are recorded while there's still a chance, it's too important to just let it die.
"Finding people who can do that are few and far between"
When I was growing up, we had to learn assembly to do anything meaning full on a C64. And 64k is all we had, and for the most part all we were ever going to get. How will the industry deal with people who can't hack/program at a low level?
Its kind of scary when you have people with that much knowledge and experience leaving the industry. Especially one as young as ours.
I always wonder what will happen with the next generations of programmers coming out of school - most of them get started with Python, Ruby or Java. And low level systems programming is mostly an after thought or an annoying class for most students.
Congratulations to Mr. Zottarelli and a great and important career. And hopefully he will write a book about his work to help us along.
I had an ex-electrical engineering girlfriend who studied assembly as a core part of her degree. I think the low-level is covered more than you think, not in software engineering but in electrical engineering where the focus is on building platforms (transistors, CPU, Ram, hardware and assembly) to enable software to be written.
Yes. Even part of my pretty standard CS curriculum included electrical engineering (breadboard to ALU), assembly (self hosted interpreter/vm) and language theory (C clone) courses. With a bit of time I or my classmates could build a computer from scratch, but the relevancy of this to my day job of many years as web and scientific programmer has been zero to minor. I think it gives me enough insight into many phenomena, but the lower, the less it is relative to making stuff actually happen for people. We are building up.
I don't think the problem is that they cannot find anybody who can program assembly in general, but they use a very exotic, old architecture with very limited memory. So someone has to learn this specific architecture with apparently limited documentation (and a google search won't cut it). And on top of that in a very unusual and critical application i.e. if you brick the processor you have a real problem. If this was e.g. a modern run of the mill ARM processor, you can find a lot of qualified assembly programmers.
With the growth of any science and industry, there is depth of specialization. The overall expertise is not decreasing, it is increasing, and the breadth has to give.
First off, this is a beautiful article. Reading about software/devops at NASA is beyond fascinating to me, and it's crazy some of the things these guys have to do; most devops are just shaking off their training wheels, and NASA is riding a rocket powered motorbike down a ramp and over 50 cars while changing the bikes tires mid-air.
But I want to reflect on something. The fact that this system is not maintainable outside of the last engineers lifetime is a failure of documentation and proper organization. I'll cut them some slack for launching a washing machine through space, but the maintainability failure is their fault; not simply the result of age or language age. Moving offices didn't help, but if you've just got papers everywhere that are so easily lost every time you move, that doesn't reflect well on the organizational front.
It also speaks to how approachable a codebase should be, and what the long term benefits of proper documentation are. You never know how long the codebase you work on will be operational. All engineers should self-absolve themselves of the hit-by-a-bus problem.
Still, I expect they'll learn from this. And with computers now it should be harder for them to actually lose things now. I just wanted to reflect on what the article meant to me.
It probably isn't their fault. I don't think anyone has managed to keep all the documentation for a engineering project of this size maintained and organised for such a long time. The people who understand how the documents are organised leave or forget things, the documentation on the documentation goes missing or gets outdated if it ever existed, stuff is misfiled, etc. Computers don't help - there's no computer system with a lifespan long enough, and every migration is a chance to lose or misplace items.
It IS their fault. it's not a natural disaster, it's man-made. That said, I'm not trying to pants NASA: it's always about tradeoffs, and whats noting is that this is a tradeoff they made, and this is what happened. Nobody died, so it makes for a good learning experience.
It it clear that the system is not now maintainable? They assigned a younger engineer to work with the man, and "impossible task" could be an exaggeration for "difficult" or could mean "never as good as the original guy", both of which are different from "he can't do it."
It might be their fault, but everyone who worked on that project in software were the pioneers at the time. So they didn't have the processes or knowledge to even know to document things.
They weren't babes in the woods - processes existed even 'way back then'. They were a government project for heaven's sake! Other departments documented every iota of information about their projects (military, aeronatics, intelligence and on and on). Its amazing NASA got away with this crap for so long. I have no idea why.
"Its amazing NASA got away with this crap for so long."
This is unfair. The popular article we're talking about is trying to dramatize a milestone, but it does not say that his retirement threatens the spacecraft. It's being managed by training someone new. When you retire after many years of service, people will make laudatory comments, say you will be hard to replace (which will be true), and then move on.
Incidentally, I believe he's not fully retired, because he's still listed in the employee directory here. Maybe it's set for the current work week, which for him, ends Friday.
Programming this thing should be an unique thing. Voyager 1 and 2 (and earlier Vikings) were not microprocessor-based. It was designed with custom-made radiation-hardened RCA CD4000, coupled with standard TTL ICs.
"Though NASA found a younger engineer who was brought on to work with Zottarelli for a year and eventually replace him, Dodd said it's an impossible task.
No one will replace him,' Dodd said."
It's an amazing career and an honor to push mankind to the farthest reaches of space. Maybe we all can stand up for a moment and clap for him and his colleagues for a job well done.
Wow. I have a PDP-11 manual with Voyager logos on it that I grabbed from a retiring physicist in the lab I used to work at – I guess I didn't realize that his retirement was part of life and reality where people move on :(
Cool manual though! I've been tempted to donate it to either the computer history museum or NASA AMES (where it seems to have come from), but I kind of feel like they have dozens of these already
"Finding people who can do that are few and far between"
I always wonder what will happen with the next generations of programmers coming out of school - most of them get started with Python, Ruby or Java. And low level systems programming is mostly an after thought or an annoying class for most students.
How will the industry deal with people who can't hack/program at a low level?
There are still plenty of people out there that can work at the low level, and young people that can work at low level too. I compare it to the beginning of CS to now. In the 70's a computer scientist was familiar with electronics, and a lot of very low level things that would be considered EE now. This was needed in the days where machines were bespoke, and coding was very bare metal. But then people built standardized hardware, and developed standard operating systems that allowed people to work with higher level languages and trust the low level to others.
This is still the case. I am familiar with assembly and X86 but I never work with it. I do not need to, there are people that work on that. That does not impair me, it is just that the world of tech has grown much larger and there is a lot to know where nobody can know it all. Some have the feeling that if you cannot work with assembly or write compilers or device drivers you somehow don't have your chops.
The skills that are "few and far between" are those wanting to work on a unique system where that knowledge is not directly applicable to today's less bespoke technologies.
Not understanding it, however, would perhaps have impaired you in the long run.
Low level programming is easier than many suggest, but at the same time more essential than many would admit. It's not that you have to practice it, but understanding it is important.
E.g. I can't help but be alerted of career programmers who have no grasp of basic things like memory pointers. This surprisingly often goes together with not understanding high level concepts like asymptotic complexity, explained away with the same argument that "you don't need it in real life, just call sort()". It's like a plumber who seems to follow the build code doing all pipework inclinations by the book, but fails to recognize the general principle that water flows downwards. It's painting by numbers at best, and religious practice at worst.
There is no room for magic if one is serious about their trade.
>> How will the industry deal with people who can't hack/program at a low level?
Assembly programming is still being taught as a EE/CS curriculum, so the younger folks know (some) assembly as well.
That said, not all programming jobs require a low level programming knowledge. Now why would a Java or Rails programmer have to know ARM assembly, for example?
The article also points out the extreme importance of documentation, because we never know what systems will still be running, or programs still in use 40 years in the future. It's truly terribly wrong to "take secrets to the grave", knowledge should be shared, we all have a stake in it.
I fervently wish Voyager's secrets are recorded while there's still a chance, it's too important to just let it die.