Monday, June 27, 2005

No Silver Bullet, April 1987

No Silver Bullet: Essence and Accidents of Software Engineering by Frederick Brooks was published in Computer magazine in April 1987, just as I was graduating from college. I stumbled upon this paper recently and read it again. It is interesting to see how little has changed in the intervening 18 years.

It was fun to read a paper written in 1987 about the future of programming/software development. It is amazing how accurate/applicable it is today still.

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

Isn’t that the case – we can make programming inexpensive, but software development is always hard.  Not because writing software is hard (it takes training, time, skilled labor) but because specifying what you want unambiguously, clearly, concisely is really hard.  And as the complexity, the scope of the systems grow beyond what a single individual can hold in their head – it gets nightmarishly hard.

In the section on “Hopes for the Silver”, I laughed at this quote:

Nevertheless, Ada will not prove to be the silver bullet that slays the software productivity monster. It is, after all, just another high-level language, and the biggest payoff from such languages came from the first transition  the transition up from the accidental complexities of the machine into the more abstract statement of step-by-step solutions.

Now, I wasn’t laughing at the fact that people thought Ada would be a silver bullet (they really did).  I wasn’t laughing at the Ada language (it was quite good actually – PL/SQL looks a ton like Ada).  I was laughing at the fact that if you replace Ada with a somewhat popular language of today – the statement still stands!

Object-oriented programming. Many students of the art hold out more hope for object-oriented programming than for any of the other technical fads of the day…. Nevertheless, such advances can do no more than to remove all the accidental difficulties from the expression of the design. The complexity of the design itself is essential, and such attacks make no change whatever in that. An order-of-magnitude gain can be made by object-oriented programming only if the unnecessary type-specification underbrush still in our programming language is itself nine-tenths of the work involved in designing a program product. I doubt it.

I doubt it too.  I remember in 1987 – people were thinking that software development would not exist as we know it in the 2000’s.  All of the software that ever needed to be developed would have been developed and we’d cobble together applications by assembling components.  We wouldn’t have programmers, but more of an army of assemblers, tying together all of these pre-existing bits and pieces into applications on the fly.  It didn’t really happen.  In fact, I see little real world reuse of software.  Sure, we have class libraries in the millions, but they are little more than the utility packages I used to develop in the past (we called it modular coding back then).  They solve bits and pieces of a problem but are not major components.  Maybe it is the environment I work in, but I don’t see huge amounts of reuse happening, not with business components anyway.

Artificial intelligence. Many people expect advances in artificial intelligence to provide the revolutionary breakthrough that will give order-of-magnitude gains in software productivity and quality. I do not. To see why, we must dissect what is meant by "artificial intelligence."

Ah, AI, it was going to save the day.  It along with Expert Systems and Natural Language Processing.  Any software worth its bits and bytes had the AI or Expert System tag associated with it.  Now, like CASE, people distance themselves from those terms, as the end product was so far removed from the promise.  Expert Systems actually worked though, they were just oversold.  They still exist today, you might even say there is an expert system in the database – it applies the ROT, rules of thumb, to the metrics and making sometimes reasonable suggestions based on it.

Graphical programming. A favorite subject for PhD dissertations in software engineering is graphical, or visual, programming the application of computer graphics to software design. Sometimes the promise held out by such an approach is postulated by analogy with VLSI chip design, in which computer graphics plays so fruitful a role. Sometimes the theorist justifies the approach by considering flowcharts as the ideal program-design medium and by providing powerful facilities for constructing them.

Nothing even convincing, much less exciting, has yet emerged from such efforts. I am persuaded that nothing will.

He got that right.  Back then, it was all about the exciting IDE’s that were coming out.  The integration of CASE with the programming environment.  Boxes, arrows, flows, automatic documentation, and so on.  Problem was, we couldn’t describe the problem we were trying to solve and it didn’t matter how pretty of an environment you had.  If you don’t understand the problem, all of the pictures in the world won’t help you.

Buy versus build. The most radical possible solution for constructing software is not to construct it at all.

That guy was pretty forward looking for his time.  Not many of the application vendors you know of today even existed back then.  The stuff we take for granted today, didn’t exist.  It was just being started, just being formed.  I think he got that one right, dead on.  There is the real code reuse, forget OO, just buy the stuff.

Anyway, it is an interesting read if you have the time.  If you were in the industry back then, it is a good “time travel” paper (see table 1, “Exciting vs. useful but unexciting software products”).  The main gist is at the bottom:

My first proposal is that each software organization must determine and proclaim that great designers are as important to its success as great managers are, and that they can be expected to be similarly nurtured and rewarded. Not only salary, but the perquisites of recognition office size, furnishings, personal technical equipment, travel funds, staff support must be fully equivalent.

Great designers, not coders.  People who can put it all together.  I think he got that right (but we still aren’t there yet).

POST A COMMENT

31 Comments:

Anonymous Justin Rowles said....

We wouldn’t have programmers, but more of an army of assemblers, tying together all of these pre-existing bits and pieces into applications on the fly.

But but but... that's pretty much Java (or C# for the MS boys). I sure as hell never write an actual memory-munging bit of code if I can avoid it. I never printf ATX commands into a modem any more, do I? The tinkertoys approach to programming did arrive and did avoid one set of programming errors. We just still have to work out how to deal with a customer who says "I want it to work just like it does now, but better".

Mon Jun 27, 10:09:00 AM EDT  

Blogger Thomas Kyte said....

Justin Rowles said...

You wouldn't be writing java at all.

You'd pick up a person object, glue it to a scheduler object, and a payroll object and have an HR system.

That you don't pump data out to a modem doesn't mean building software systems is easier (heck, it is probably easier to learn the AT commands than to master the "modem class").

We haven't arrived anywhere close to the tinkertoy approach at all. Things are not significantly different from what they were 18 years ago. You have "utility libraries", I had utility libraries. They make accessing modem easier, but they don't do anything "business" oriented.

Java, to quote the article, It is, after all, just another high-level language

Mon Jun 27, 10:18:00 AM EDT  

Anonymous Chris Neumueller said....

A funny coincidence, I re-read NSB last weekend. You might want to buy Brooks' "Mythical Man-Month", 20th Anniversary Edition. MM-M is from the 70ies, but still a very good read. The new edition also contains NSB and follow-ups to both NSB and MM-M from his 1995 perspective.

Mon Jun 27, 10:18:00 AM EDT  

Blogger Vlad said....

They published a 20th anniversary edition of The Mythical Man-Month book in 95 in which Brooks reviewed the points from his original essays (including Silver Bullet), and argued whether they apply today (in 1995). His own conclusion was the same as yours - not a lot has changed in Software Engineering in the past 20 years.

If I remember correctly, he just didn't expect GUI shells to become that popular.

Mon Jun 27, 10:20:00 AM EDT  

Anonymous Anonymous said....

This is the best article I ever read on software project management. The author compares failed software development projects to a failed ship building project from 400 years ago.

http://www.cse.ogi.edu/~dfairley/The_Vasa.pdf

Ryan

Mon Jun 27, 10:35:00 AM EDT  

Anonymous Dan Kefford said....

Fascinating read... some of the things I agree with, some of them I don't, but very interesting nonetheless.

Graphical programming

From the Brooks article:

Sometimes the theorist justifies the approach by considering flowcharts as the ideal program-design medium and by providing powerful facilities for constructing them.

Nothing even convincing, much less exciting, has yet emerged from such efforts. I am persuaded that nothing will.


... and from Tom's post on this issue:

He got that right. Back then, it was all about the exciting IDE’s that were coming out. The integration of CASE with the programming environment. Boxes, arrows, flows, automatic documentation, and so on. Problem was, we couldn’t describe the problem we were trying to solve and it didn’t matter how pretty of an environment you had. If you don’t understand the problem, all of the pictures in the world won’t help you.

I dunno... I frankly could not think of designing software today without using GUI tools to _some_ degree: MagicDraw for UML, SilverRun for ER design, JBuilder for some amount of code generation and refactoring.

Is it that Brooks is pointing out the limitations of the hardware at that time on which such software would run:

Second, the screens of today are too small, in pixels, to show both the scope and the resolution of any seriously detailed software diagram.

... or that some tools/methodologies were simply too complex in and of themselves:

If one superimposes all the diagrams generated by the many relevant views, it is difficult to extract any global overview.

(I wonder if this is a jab at UML, by the way.)

Otherwise, I think GUI tools _can_ lead to increased productivity if the tools are chosen wisely. That means that the developer must understand a particular tool's limitations, as well as its features, needs to remain focused on the main objective (building a quality product), not fight with or get lost in the tool, and must understand the fundamental concepts of software design to begin with.

Great designers

From the Brooks article:

How to grow great designers? Space does not permit a lengthy discussion, but some steps are obvious:

* Systematically identify top designers as early as possible. The best are often not the most experienced.
* Assign a career mentor to be responsible for the development of the prospect, and carefully keep a career file.
* Devise and maintain a career development plan for each prospect, including carefully selected apprenticeships with top designers, episodes of advanced formal education, and short courses, all interspersed with solo-design and technicalleadership assignments.
* Provide opportunities for growing designers to interact with and stimulate each other.


Wow... he said this in 1987? And for the most part, it's still not being done. I'd like to work for this guy. ;P

I think these maxims apply to just about _any_ trade, but they are especially critical to developers because of the intellectual nature of their trade and because, for better or for worse, technology changes every single day.

Mon Jun 27, 10:40:00 AM EDT  

Blogger Bill S. said....

Great article there - brings back memories.

Tom Kyte wrote:
"They still exist today, you might even say there is an expert system in the database – it applies the ROT, rules of thumb, to the metrics and making sometimes reasonable suggestions based on it."
You gotta watch that stuff, Tom. I smell a new article coming on "that other" site quoting this little snippet.
:-D

Mon Jun 27, 10:46:00 AM EDT  

Blogger Thomas Kyte said....

http://www.cse.ogi.edu/~dfairley/The_Vasa.pdf

Ryan


Yes, that was good. Liked the writing and the message.

I like the analogies, for some reason it made me think of this:
http://www.utc.com/press/highlights/2003-08-27_tower.htm
which was just posted to the blog over the weekend.

Look at the stuff about testing, seems our industry isn't any different from anything else :) From ship building to elevators, we have the same problems and solutions.


Dan Kefford said...

Fascinating read...


As for the graphical programming, I think it has helped incrementally. It is a step, but a small one. In the past, we had screens of almost infinite size (we called them plotters, it was on paper :) But the size didn't matter, it just got so complex. The GUI's can help organize, they are useful, but they are not a silver bullet.



Bill S. said...

Great article there - brings back memories.

... and making sometimes reasonable suggestions based on it."


You did note the emphasized text, the wording there was very intentional :)

And -- if the software can do it, does do it, why do we need to redo it ;)

Mon Jun 27, 11:02:00 AM EDT  

Anonymous Anonymous said....

I’ve worked in software development for almost 20 years. My experience has been that projects don’t fail because of the technology used. They fail because there is inadequate or no analysis done for requirements, design and programming specifications and testing plans. It was that way 20 years ago and it is still the same today.

Mon Jun 27, 11:22:00 AM EDT  

Blogger Tim... said....

RAD and iterative development methods have pretty much killed design skills as far as I can see. Not that there is anything wrong with these as concepts. As always the problem is their implementation. A large number of development projects take the view that iterative development means you don't have to think at all, just start. I hear quotes like, "We'll refactor that later!", all the time. The problem is that later never comes.

I repeatedly think back to my early days in IT working on real-time control systems with Oracle as a back end. We created requirement specs, functional specs and module specs, all of which were signed off by the customer. We then designed our DB to support the requirements. We wrote pseudocode, had it reviewed, then turned it into real code (PL/SQL and Pro*C). More often than not it worked first time due to the details and care taken along the way. when it was over the customers liked what they were given because they were involved in the whole process.

Fast forward a few years and what do we have, a spec in a single email that the customer hasn't approved, but, "It's OK! We can change it later. It's iterative development!"

Progress is a wonderful thing.

Tim...

Mon Jun 27, 11:53:00 AM EDT  

Blogger jimk said....

WHat is killing the growing of great designers today is the desire for immediate gratification.

Mon Jun 27, 12:02:00 PM EDT  

Blogger Jeff Hunter said....


All of the software that ever needed to be developed would have been developed and we’d cobble together applications by assembling components. We wouldn’t have programmers, but more of an army of assemblers, tying together all of these pre-existing bits and pieces into applications on the fly.

Depends on the shop. The last place I worked we spent about 75% of our time integrating data from several "best of breed" packages that ran customer billing, CRM, ERP, telco provisioning, and switch integration on all different platforms and databases. In my current gig, we integrate with off the shelf ERP, but most of the software is written in-house for business advantage.

Mon Jun 27, 12:30:00 PM EDT  

Blogger Bill S. said....

Tom Kyte wrote:
"You did note the emphasized text, the wording there was very intentional :)"

Yeah, I noticed. But you also realize that THAT part is the part that will get left OUT of the out-of-context quote, right? ;P

Mon Jun 27, 01:13:00 PM EDT  

Blogger Joel Garry said....

I know! Let's use a COTS system! http://mae.pennnet.com/Articles/Article_Display.cfm?Section=Articles&Subsection=Display&ARTICLE_ID=96013&KEYWORD=%22smart%20ship%22%20postmortem

Mon Jun 27, 03:48:00 PM EDT  

Anonymous Arian said....

Joel Garry said...
I know! Let's use a COTS system!

I love that word. In Dutch we have the word KOTS (pronounce it the same way). It means 'puke'.

But I do make a living with supporting Oracle E-Business Suite.

Ok. I'm off to go and see a psychiatrist.

A.

Mon Jun 27, 06:58:00 PM EDT  

Anonymous Doug C said....

The very last quote you had from that article rang a bell.. I knew I had read it recently. It is republished in a good book called "Software Project Management - Readings and Cases" by Kemerer. It is a very good book although it's not they type of book you just sit down and read front to back. I sometimes wonder if the reason great designers aren't identified is because the manager types who should be identifying them really lack the background to spot them in the first place. I can get a sense whether I'm working with a great developer in just a few conversations - are they forthright and willing to answer questions; do they seem driven by an inate love of what they are doing, etc., Most manager types I know just ask about deadlines and progress and how long will it take and what can we do to help (hire more people) etc., and they can't distinguish the real talent in the first place.

Tue Jun 28, 02:35:00 AM EDT  

Blogger Tim... said....

Design and development skills have been undervalued for years. Most techies are faced with the choice of moving into management or going freelance if they want a pay rise.

In my experience replacing a decent manager is far easier than replacing a decent designer, DBA or developer. If my experience is common to others, why is it that these skills are undervalued?

I can only assume it's bad management ;)

Tue Jun 28, 05:29:00 AM EDT  

Blogger Tim... said....

Just to clarify. My first post regarding iterative development was meant to convey my opinion that it is a step backwards, not forwards.

On reflection my closing statement should have read,

[sarcasm]
Progress is a wonderful thing.
[/sarcasm]

Is there an emoticon for sarcasm or rolling of the eyes? :)

Tue Jun 28, 06:24:00 AM EDT  

Blogger Jeff Hunter said....

[sarcasm]
Progress is a wonderful thing.
[/sarcasm]

Is it really sarcasm if you have to point it out? ;)

Tue Jun 28, 08:30:00 AM EDT  

Blogger Renee said....

Tim, management is where good techies turn into pointy-haired bosses. My last boss, who hasn't written much sql in years still bows to the UNION god while team moved on to great stuff like DECODE and CASE. Aye carumba! ;)

I wonder what the article's author, Frederick P. Brooks, Jr., is doing these days.

Tue Jun 28, 12:41:00 PM EDT  

Blogger Thomas Kyte said....

I wonder what the article's author, Frederick P. Brooks, Jr., is doing these days.

http://www.cs.unc.edu/~brooks/

Appears to be still very active in the industry.

Tue Jun 28, 12:50:00 PM EDT  

Blogger DaPi said....

Tom said: If you don’t understand the problem, all of the pictures in the world won’t help you.

If you continue not to understand, that's true.

But I'm a firm believer in diagrams (entity-relationship, state-transition, flow chart, data-flow . . ) as a way to investigate & communicate about the problem. The white- (or black-) board is powerful tool. In some ways the GUI tools make it too formal - some good doc I have are scanned scraps of paper with hand-drawn diagrams.

The least useful doc I have is a computer generated E-R diagram: fills several walls, the text approx 2pt. Echos of the Microsoft Helecopter joke.

Tue Jun 28, 12:56:00 PM EDT  

Blogger Thomas Kyte said....

DaPi said If you continue not to understand, that's true.

I guess in re-reading that, my intention was not clear.

If you don't understand the problem.

No number of pretty pictures drawn by you (the person without an understanding of the problem) will be useful. You will have drawn a picture of something that solves (maybe) some problem but not the problem at hand, because you don't understand it.


But pretty pictures from a well understood problem can be very useful in conveying that knowledge (the understanding) from generation to generation.


Point was -- without an understanding of the problem at hand, no amount of tools will help you.

Tue Jun 28, 01:10:00 PM EDT  

Blogger Tim... said....

Totally Off Topic:

Have you altered your picture Tom? It no longer seems to shrink so well.

Tue Jun 28, 02:15:00 PM EDT  

Blogger Thomas Kyte said....

tim said totally off topic

Yeah, I was using the flickr copy of it -- but used the "m" medium one -- now I'm using the "s" small one and it scales right ;)

thanks

Tue Jun 28, 02:35:00 PM EDT  

Anonymous Gabe said....

Dan Kefford said:

How to grow great designers?
[list …] …
Wow... he said this in 1987? And for the most part, it's still not being done.


That maybe was somehow applicable/possible 20 years ago with small development shops and the exuberance of new things (the computer revolution … or rather its spread). Nowadays, it would be the exception to the rule to have something remotely akin to “recognising, nurturing and retaining” talent. It just doesn’t happen (and some dry HR policy is no substitute) … I doubt it will be any different in 20 years time. What I see happening right now is, at best, a stalemate between the corporation and individual: as long as the individual is allowed to carve a niche for him/herself then there is an alliance of sorts; else, the individual moves on or strikes on its own and the corporation goes after those likely to conform (which also are likely costing less) … of course, they exhibit the same range of intelligence and human emotions … and, over time - once they pass the point where their instinct tells them to feed their families first and then voice their concerns, they will get into the same predicament.

The open source community with its inherent meritocracy system is one place to get “work” (as in effort) satisfaction … open peer exchanges (like forums) is another avenue since it is free of political conformism … it can get surgical, so it is a double-edged sword, but that’s the beauty of it … it forces one into constant self-appraisal. At the end of the day it doesn’t matter who you are (and I certaily don’t care who you think you are) … if you state something publicly and haven’t done the due dilligence then I’ll be pointing it out [assuming I can grasp the subject, of course].

Having said that (not that the “you” above was about you Tom), I don’t agree with your interpretation regarding the “great designers” …

Great designers, not coders. People who can put it all together. I think he got that right (but we still aren’t there yet).

I think the article suggests there is a challenge for organizations [“software art centers”?] to extend their effort from growing “great managers” to [also] grow “great designers upon whom the technical excellence of the products will ultimately depend”. In passing, this seems to tell a story with the organization as the reference point … in fact the success of an organization doesn’t necessarily translate into “success” of the individual members.

I don’t think it is about executives (business visionaries), managers (business doers), designers (technical visionaries) or coders/testers (technical doers) … it is about people consistently making good/solid decisions within the realm of their [appropriate] role; that is, it is about good decision makers. The number one skill of a good decision maker is to recognize when (s)he’s made a bad one and use that as a tuning mechanism.

We all likely make bad decisions at one point or another … some just more than others. The question is “do we leave our ego or desire not to acknowledge the failure stand in our way for self-improvement”? Some just never learn to lose … and are only too keen to construct another world where they were right (and hence, others were wrong) … it just clouds their judgment and brings chaos around them … no wonder the software projects they are involved in fail (it may not even “fail” from their point of view). It isn’t just about instant gratification (as jimk put it) … it is about perpetual gratification.

Then again, I could be wrong … or too cynical ... well, certainly too verbose.

Tue Jun 28, 04:19:00 PM EDT  

Blogger Thomas Kyte said....

well, certainly too verbose

Nah, I've been know to take pages to say what others might say in paragraphs myself :) And I'll keep on doing it.

Tue Jun 28, 04:44:00 PM EDT  

Blogger Richard Byrom said....

And google earth has just landed. Read about it here http://battellemedia.com/archives/001661.php

download it at http://earth.google.com

Wed Jun 29, 04:00:00 AM EDT  

Blogger Fahd Mirza said....

Thomas Kyte said...
"Java, to quote the article, It is, after all, just another high-level language"

In the preface of Practical Oracle 8i, Jonathan Lewis has said the same thing. Great writer he is, veyr lucid, very informative, though a little pompous in his writings.

Mon Jul 04, 01:04:00 AM EDT  

Anonymous S.C.Sprong said....

You are me and I claim my Delorean that goes to 88 mph...

What _has_ changed for the better in my view is that, thanks to the ubiquity of cheap, networked, stupenduously powerful computers, many small to medium scale software engineering problems have become within reach for very small teams and therefore they are finally being done _right_.

I have become a strong believer in the power of small, highly talented designer teams (BSD vs the Linux horde, PostgreSQL vs MySQL, etc.)

Fri May 12, 11:49:00 AM EDT  

Anonymous Anonymous said....

Here is the fixed link.

Thu Dec 20, 05:49:00 PM EST  

POST A COMMENT

<< Home