Newsmaker: Debating the morality behind software development

In the long history of software development--let's loosely mark the starting point around the time of the ENIAC--code writers have dealt with a myriad of technical and business challenges. It's fair to say they've also not had to confront questions of morality or ethics about how governments later deploy their finished work.

Until now.

Grady Booch, the inventor of the Unified Modeling Language, says those days of splendid isolation are--and should be--coming to an end.

Booch was the first chief scientist at Rational Software when it began in 1981. He kept the job after IBM bought the company in 2003 when he was also elevated to the rank of IBM fellow.

Maybe it's the freedom that comes with possessing a small fortune or perhaps it's just in his DNA to make waves, but Booch relishes the "voice in the wilderness" mantle--both inside and outside of the technology world's largest corporation. CNET News.com spoke with Booch about his ideas concerning software and ethics during a recent swing he made out to the West Coast.

Q: You've gone on record talking about this question of morality in software. I didn't think one could classify software as moral or immoral. What's behind your thinking?
Booch: Even though what we're doing is deeply technical stuff, there are ethical, moral implications about what we do. And it's not just in our sciences--look at the struggles the physicists of the '40s and '50s had dealing with their ability to unlock the secrets of the universe.

Even though what we're doing is deeply technical stuff, there are ethical, moral implications about what we do. And it's not just in our sciences--look at the struggles the physicists of the '40s and '50s had dealing with their ability to unlock the secrets of the universe.

You're talking about nuclear power?
Booch: I'm talking about nuclear power and not just nuclear power, but also nuclear weapons and the like. It's our ability to unlock these secrets of the universe for either good or bad.

And so?
And so there were great discussions then--and even today--to the effect that I may have the ability to do these things, but should I do these things? The same thing is true in software systems.

But those are technical issues, then.
Booch: They're not.

But they don't have anything to do with morality.
Booch: I'm leading up to where the morality issue goes up this (development) ladder. It's the place where it's not just a matter of whether we can build or want to build but also the question of whether we should build.

Here's an example. London's installing more video cameras per square mile on the street than anybody else. All right, not a lot of software there. But what happens when they couple that with facial recognition software so I can actually track individuals as they go through the city?

But that's not a question that the software developer gets presented with. That's something for the city of London to consider based upon its needs.
Booch: Yes, but at the ultimate level, the software developer can say, "Do I want to actually build a system that potentially could violate human rights?"

What software developer do you know actually thinks about that when he or she sits down at the keyboard?
Booch: I know many. There is a group called Computer Professionals for Social Responsibility where many of its members think about that kind of thing. That group was formed to deal with the social issues of the developers. Do I as a developer, coming fresh out of college, decide to go work for someplace in Silicon Valley working on a benign business application? Or do I work for some defense contractor? So that's a moral decision that a person has to make to use his or her skills.

For the sake of argument, I also could say that while Google is a for-profit corporation, it's been involved in China and some people have raised questions in connection with the company's policies. So when some freshly minted engineering candidate out of Berkeley decides where to apply for a job, does Google then get put on par with the military as far as these moral questions are concerned?
Booch: That's a decision that person has to make. The issues you raise are philosophical ones. Let's say I'm working on some bit of software that enables sort of a social networking kind of thing that enables connectivity among people and there's potential for the exposure of lots of information. Well, do I then add a particular feature realizing it may have a coolness factor. But at the same time I may just have found a way that pedophiles can get into this network more easily.

More Newsmakers

[projectnation]

Horrible Interview, Charles

I hardly ever post on here but just cannot stand it when a reporter get so belligerent with their guest. Wish I got paid to ask questions like "And so?". Anyway, great idea horrible interview.

[charlie cooper]

Um, not really

thanks for your feedback but you totally misintrepreted here. i really enjoyed my interview with grady. i'll let grady speak for himself but i think he had a good time as well. but my job isn't to take dictation. it's to challenge and probe. grady's a very smart man and he made his case eloquently. there was no disrespect intended - and i don't think he came away thinking that was the case.

[Test99.1]

Second The Comment

For an interviewer to play devil's advocate to an interviewee's position is a familiar and unexceptionable technique.<br /><br />But that was not how I read Charlie Cooper's questions. Whatever he meant to say, what I heard was aggressive and ethically challenged questioning. Sometimes you get only one chance to make a bad impression.

[Zimm2]

The EULA is about as immoral as you can get...

Morality issues start way before you start thinking about how <br />software is used. Just look at the average EULA; the customer is <br />screwed no matter how bad the software is... The average <br />developer doesn't even stand behind his or her product. That's <br />were I believe this discussion needs to start!

[baike]

ivory tower

Come back to reality. This is such an ivory tower discussion. Booch and his buddies have the time and money to contemplate 'should I', while the average developer in the US is scared his job is going overseas. Do you really believe that a developer in the US will say no to any work, when there are a billion Indians who will do the same work at half the rate (or less)?<br /><br />Also, Booch sounds pretty high on developers, as if they're turning lead to gold or something. Does he realize that whatever we code a machine to do, a person can already do? The only advantage the machine brings is speed and scale.

[bvdon]

Your fellowship has been revoked!

How did this guy get to be an IBM Fellow? His philosophy on <br />software morality is not very well thought out and he is putting too <br />much responsibility on the programmer and not the business <br />person driving the project. What world does this guy live in?

[Snewton628]

The Lack of Perspective

First, it would be more proper to describe Grady as "one of the inventors of UML", not THE. There was, as I recall, three primary visionaries involved. <br />Second, Grady did NOT represent his input as a thought-through personal answer to the question of the moral responsibility of the individual technician or developer. He was pointing out that the fact that such questions are now being asked indicate that many of our innovators have gotten past the "Gee Whiz" stage and have begun to think about the potential impacts of their efforts on society. Anyone who finds that irrelevant, or the province of the managers only should check the archives of the Nuremberg War Crimes Trials, where the German officers made that argument much more forcibly, and where it was resounding (and correctly, in my view) rejected.

[basraw]

If developer A doesn't want to build it..

1) <br />If developer A doesn't want to build it..<br />Developer B will and developer B will get paid while A goes hungry.<br /><br />"But that's not a question that the software developer gets presented with. That's something for the city of London to consider based upon its needs.<br />Booch: Yes, but at the ultimate level, the software developer can say, "Do I want to actually build a system that potentially could violate human rights?""

[wildthought]

I was only following orders...

I am not suprised by the comments that feel it should not be a programmers responsibility to think about what they are building and its impact on society. I am however dismayed. Taken to its extreme we the people who made and installed the gas chambers (pretend they were independant contractors) were very much morraly responsible for the deaths that occurred in the holocaust. <br /><br />Do we really want to live in a society where the government can and does track our individual movement through software? We of all people know how close that Orwellian nightmare is to beocoming technologically feasible. It is one thing to be aplotical and not care, it is quite another level of immorality to participate in the erosion of our civil liverties and that of others. Go Grady!

[lkrupp]

If Hitler or ToJo had the bomb...

Read the book Hiroshima Day One. The American physicists did <br />indeed discuss moral issues as to the building of the first atomic <br />bomb. All one has to do is imagine if those scientists had come <br />to the conclusion that the bomb should not be constructed and <br />had refused to do further work. Both Nazi Germany and Japan <br />had bomb projects and would have built one sooner or later. <br />And of not them then some other despot down the line. Where <br />might we be if that had happened?<br /><br />The whole interview was an exercise in futility and wrong <br />headed thinking. If developer A doesn't add the feature being <br />morally debated then developer B WILL add it. Developer A can <br />do nothing to prevent it other than run off at the mouth. <br />Splitting the atom, cloning a human being or building a <br />computer, once the genie is out of the bottle moral arguments <br />mean nothing. The best one can hope for is to control the issue <br />through regulation or negotiation.

[spothannah]

Hitler/Tojo

As the catcher/philosopher said: "It isn't over until it's over." The system is not complete and so there is no "right" answer. Maybe by our winning we are not creating a worse Hitler/Tojo. If Germany would have won WWI maybe Hitler would have been ended up an artist or architect. My point is no one knows how it's going to be so we have to act with incomplete knowledge. And then, we have to live with the knowledge that we "acted without knowing" that's the nature of this event we call life. So far, I have learned that the only freedom we have is "choice" and maybe that is illusory. However, if even "choice" is not influenced by freedom in the sense that we are "free to choose" then it appears to me that we are just part of some program that is crunching along to some "preprogrammed" answer and then what "freedom" do we have? I fear either answer.

[lingsun]

Why not question the morality of Google helping to oppress Chinese?

What they've done should be a crime.

[pwoon]

Americans are oppressed

Did you ask the Chinese if they're being oppressed? In my opinion, Americans are being oppressed, but would Americans say they are?

[spothannah]

morality

Read Kierkegaard. Do your best. Nobody here gets out alive. Love.

[R.Jefferson]

Morality and ethics are relative

The logic a business uses as opposed to a person is very different. A person may be a thoughtful and principled person on a human to human level. <br /><br />A business is principled in making money. The transactions that occur are between an impersonal group (a business) and a consumer (a person), so there?s no meaningful human interaction taking place. The profit motive is the guiding principle of the business world and morality and ethics are in direct conflict with the profit motive.

[cemptor]

The debate isn't really particular to software development

One of the worst interviews I've ever read...<br /><br />Not so much the interviewer as the interviewee. Lost some of my respect for Grady Booch, who makes vague allusions to morality, without expanding on what he means...<br /><br />The debate isn't really particular to software development as it is to any profession. Do you believe in the end result of what you do? Can you connect the dots and understand the ultimate result - expected and side effect (as in pollution, social consequences, etc.) of what you do.<br /><br />It does need to be raised for software developers too, who more often than not, are more focused on their methods and tools than the end goals. I know lots who are brilliant at programming, but couldn't care less about what the business goals of the software they write are.

[vlgreenleaf]

Interesting, coming from someone who....

Started out in the Air Force..and made much of his fortune because of defense. He can speak from experience.

[mr e]

balderdash!

ethics doesn't come into it, this is just a way of making software engineers think they're more than mere number crushers. the final comment that being in the business is 'cool' is a giveaway - the interviewee simply wants to inflate the importance of himself and his co-workers in the industry, to render something dull and nerdish as interesting, hip and significant. either you're a 'moral' person or you're not, and there are many degrees and definitions of morality. if you adopt a strictly ethical approach you absolutely won't work on certain things that go against your personal 'code', but - as someone has already pointed out - if YOU don't someone else certainly will, and you'll go hungry. naturally cnet, which feeds off the software industry and shares its insecurities would like to inflate its importance, and enhance their own 'standing' by association, but that doesn't mean this is a question worth thinking about for more than 2 seconds.

[gbooch]

Feedback

Not surprisingly, reaction to my interview was polarized; this is <br />an emotional topic. That being said, i must commend <a href="#">t<br />his reader</a> who really grokked my point. While some of the <br />comments posted in reaction to the interview accused Charles of <br />being belligerent, I never saw it that way: this is an important <br />and little-discussed subject, and Charles asked me some good, <br />hard questions. Thank you, Charles.