Software-Quality Discussion List
Digest # 013

Tenberry Home
Software-Quality Home
Reducing Defects
Automated QA
Site Index

           Software-Quality Discussion List

      S O F T W A R E - Q U A L I T Y   D I G E S T

      "Cost-Effective Quality Techniques that Work"
List Moderator:                      Supported by:
Terry Colligan                       Tenberry Software, Inc.     
March 27, 1998                       Digest # 013



    Status Report

    Helpful statistics

    ==== CONTINUING ====

    Re:write testing logic right into the product
      John McGrail 

    Re: Software-Quality Digest # 011
      Jon Chappell 

    OS testing (Mark Wiley)

    Re: OS Testing
      John Cameron 

    Re: attempts at controversial statements
      John McGrail 

    ===== NEW POST(S) =====

    self diagnostics.
      Travis Siegel 

    Re: attempts at controversial statements
      John McGrail 

    What books have already been reviewed?
      Rita Williams 

    ==== BOOK REVIEW ====

    (none)   (What?! Again? ;-)


  Status Report

  I have gotten a few reports of doubled issues, many more of
  no problems with the last two issues.  I think my poor list
  server (NTList & NTMail on a 486/66 connected to the Net
  via a measly 56K leased line) is overworked, and not
  completely defect-free, either! ;-)

  I'm trying a slightly different approach to this version,
  so please let me know if you receive two copies, with the
  second having most equal signs changed to =3D.

  Helpful Statistics

  One of the things I had hoped would happen on this
  discussion list has started -- people are reporting statistics
  about their systems, such as test coverage, frequency of
  testing, etc.  I am extremely pleased, because data of this
  type is not generally available.  I have one request, though:

    Please add your opinion or belief about how well the
    technique worked, and whether you would recommend more,
    less, or the same.

  For example, reporting that you spend 5% of code on logging
  is helpful, but adding the fact/opinion that your error rate
  was cut in half would be fabulous!  And don't just report
  things that are successes, either -- we can probably learn
  more from things you've tried that didn't help much!

==== CONTINUING ====

++++ New Post, New Topic ++++

From: John McGrail 
Subject: Re:write testing logic right into the product

>  I'm curious -- do you see any benefit from having so much
>  logging code?

Yes.  Definitely.  When things are working right in the test
environment you (the tester) get warm fuzzy feelings when you
see the right things flash by in the log file.  If things
aren't flashing by when they should be, you start
investigating whether it's hung or just thinking.

And when helping other testers, documenters, bug fixers, etc -
generally people who have a limited understanding of the
overall system and they just want to make _this_ piece work.
Often times I find they've not set something up properly.  A
quick look at their log file usually confirms this.

As I mentioned in the original posting, it has also been
useful when helping solve customer problems.  Both when there
are actually bugs in the product, and when the customers are
doing not quite the right thing.

This is probably a whole topic in and of itself, but ... I
think a high level of logging is more important in an
object-oriented environment (which the current company is and
the previous company mostly wasn't).  And, I think the logging
capability is more useful in server oriented software than in
client oriented software.  When something goes wrong, the user
can usually show you what they did.  When something goes wrong
in the server, it's not as easy to replicate.

> I thought you were saying so up until the last paragraph,
> where you imply that both companies have a similar level of quality.

Aah, subtlety is my friend ...  I said the two products have a
high degree of testability with a minimum impact on the source
code.  But, that in no way implies that the products have a
similar - or even a high - level of quality ;-)  Just because
large portions _could_ be tested with external tools doesn't
mean we do so.  Resource constraints prevent that at the
current job and prevented that at the last one.

>  Can you contrast the two companies, from a quality and
>  reliability point of view?

I would say in the gui product functionality testing was more
important. With the network server products, I'd say
performance, and reliability under stress testing are more
important than functionality testing.

I would consider the GUI product had medium quality and
reliability in some areas and high quality/reliability in
others.  That was because we concentrated testing efforts in
certain areas.

For example: Very few bugs would be reported by customers
against the database subsystem and the tuxedo (or 3tier)
subsystem.  Most bugs in those areas were reported against
platforms or databases we didn't test against.  85% (or more)
of our customers bought licenses for 3 databases on 2
platforms - so those got the most coverage.  The cost to cover
the other 30 databases we supported on all 20 or platforms we
supported would have been prohibitive.

We did minimal testing/verification on all other
platforms/dbs.  We had two testing organizations: The QA group
- which did depth testing of certain features on the core
platforms.  The Product Integration group did breadth testing
(very smoke like tests) of most areas of the product - after
installation are all the expected files there, are there no
extra files, can the user compile their app, connect to the
DB, build a simple form, query the DB - that sort of stuff.
Prior to the extra group, we had many support calls from
customers because a tape didn't work, or a simple query on an
odd platform would crash the product - highly visible, very
embarrassing, easy to fix problems.  And, easy to prevent as
well - thus the creation of the group.

While the group was originally created to test media before it
was shipped to the customer it became involved in many places
in the release process over time.  Such as running their
battery of tests on each build before QA got a hold of it.
Basically, acceptance testing for QA.  But, I digress.

I only ran code coverage a couple of times over the 5 years I
was there.  At best we got about 45% coverage of the entire
product with all the tests (manual and automatic).  However,
over 70% of the database interface code (that we intended to
cover) was covered with automated tests.

As for the present company ... There is a high degree of
quality and reliability in the servers.  Less in the GUI
configuration tools.  This is due to a combination of things:
the servers MUST - ah that word I dislike - be reliable, the
problem space is well-defined, good automated testing and,
incredibly sharp and thorough developers.

Our users are more tolerant of problems in the gui than they
are of problems in the servers.  Thus we've concentrated more
of our efforts in testing the servers.  If I had to guess, I'd
say our automated tests cover 20 to 40% of the code.

And, while code coverage is a good way to measure how much of
the product code is exercised by the tests, I'm not
disillusioned into thinking that 100% code coverage = 100%
test coverage.  Even if 100% code coverage were an attainable

John McGrail
American Internet Corporation -
1 (781) 276-4568

++++ Moderator Comment ++++

  Thanks for the clarification.  While I'm bubbling over with
  questions of effectiveness measurements, since you seem to
  have more actual measurements than most folks, it occurs to
  me that what I'd really like to know, about error rates, etc.
  would possibly put you in the position of "airing dirty linen".

  So I won't ask...  At least not you, specifically!

++++ New Post, New Topic ++++

From: Jon Chappell 
Subject: Re: Software-Quality Digest # 011

> ++++ Moderator Comment ++++
>   This seems like a great idea to me!  We use built-in and
>   automated diagnostics for the new software that we develop,
>   and find it unbelievably valuable, particularly when
>   coupled with daily running of the diagnostics.
>   You are using the same "daily running of automated
>   diagnostics" to insure the quality of both an operational
>   system and the data itself!
>   I'm not really building or supporting those kinds of
>   systems, but it seems like it would be very effective.
>   What do our readers think about this?

This is a very low-cost, practical thing to do. I have some
suggestions for bringing up the diagnostic and database
integrity levels:

1) If you can arrange for it, have the log file written to a
network server. If your applications run on several PCs on the
same LAN, this gives you a central place to audit when you
have to troubleshoot. Make sure you hang a "client ID" in
there somewhere, so that you can tell which user on which PC
sent which log message (at what time and date). Once this is
set up, you can do a "system health audit" from a single
administrative workstation, which you might even access
remotely for quickie maintenance.

2) Regarding the database index rebuild. Consider having a
program that runs at night or during the weekend, which wakes
up periodically and rebuilds indices. This can be conditional
(e.g. rebuild once a month whether you need it or not,
otherwise, rebuild only if an error has been detected).

3) You can also collect "stats" while you are doing this, and
have the administrative program log record counts, disk space
free, etc. Depending on the situation, this can help you (or
someone tasked with checking on the system periodically) know
about a problem before a bona-fide crash.

4) Taken to the extreme, this log & stats data could even be
configured to automatically get emailed to your own site,
providing an auto-monitoring service. Depends on what level of
service you want to provide.

> Topic 2 - attempts at controversial statements.
> ----------
> Please agree with/disagree with/ignore the following
> statements.
> 1. Every production program is granted a budget of at least
> 5% of code volume, 5% of data volume and 5% of execution
> time reserved for internal testing, consistency checks,
> post-production debugging and similar functions (which are
> not required to contribute at all to the primary purpose of
> the software).

I would not limit myself to 5%. A better rule is: Put in
whatever is *necessary* to achieve a good level of diagnostics
and consistency checks and count that as part of the
operational cost of your software. If, for some reason, proper
diagnostics in a given application costs you 12%, then so be
it, so long as it still meets any real-time requirements.

It is better to have an application that you can trust is
correct than one that runs 10% faster which you cannot trust.

Re: compliance. The dev staff I work with lives this one. We
accept this as necessary for maintaining a 24x7 system.

> 4. A programmer (or team) responsible for designing and
> implementing a function (or program or module) must also be
> responsible for designing and implementing test procedures.
> QA is responsible both for measuring the accuracy and
> coverage of test procedures, and (by running the test
> procedures) for measuring the degree to which the function
> is in compliance.

IMHO, programmers are responsible for unit testing during
development, which means "prove it as you go along". Design
and implementation of test procedures should be a joint
venture between developers and QA staff. That way, the result
is less "designer-centric", and hopefully easier to understand
than a test designed by "project insiders".

-Jon Chappell

++++ Moderator Comment ++++

  I think I agree that 5% seems a bit stingy.  We have been
  quoting these percentages not as an absolute goal to work up
  to (nor to work down to!), but rather to try to give others
  a feeling for what has worked.

++++ New Post, New Topic ++++

From: (Mark Wiley)
Subject: OS testing

>   Although performance curves with "knees" or "elbows" in
>   them can provide counter-examples, can you really say that
>   anyone would notice if your system ran 3% slower?  If you
>   could achieve 10X reliability, wouldn't all your users
>   sign up in a flash?

And if we could sell the nodes for a dollar, that would be
even better. However, I don't think we can do that :-). But
your point is valid in that it is justifiable to have some
kernel code used for asserts and such in order to make it
easier to find problems (and we do that now). But the question
is how much should you do and I don't have a good answer for
that now. Perhaps after I deal with the more immediate
problems ....

>   I understand that a megabyte is a terrible thing to waste,
>   or at least to talk about wasting, but you get something
>   for spending that megabyte.  (Besides, it's probably much
>   less than a megabyte!)

Might be more, might be less, who knows?

>   By "in the large", I simply meant that you are working
>   on large, or very large, systems, while John Cameron is
>   working on much smaller systems.  Nothing more subtle than
>   that.
>   In spite of the size differences, I suspect you and John
>   and everyone else who deals with hardware interrupt stuff
>   have a lot in common.

Yes, I think John and I do have a lot in common, and I enjoyed
reading his post.

>   As the story was told to me, when this torture test was
>   first unleased on Linux, the system only lasted an average
>   of 10 seconds.  Now, each new version has to run multiple
>   torture testers for more than a day before the version is
>   allowed out.
>   An interesting, and possibly true, addendum to the story
>   is that some soul at Sun tried the torture test on Solaris.
>   That version of Solaris lasted 10 seconds before crashing!

When I started developing my high stress testing techniques at
Sequent I got similar results. And when I started the high
stress testing here the same thing happened. But it's harder
to crash or hang the system these days.


++++ Moderator Comment ++++

  I may never live down my "It's only a megabyte comment"!

  As a defense, I would say that I don't believe or recommend
  that you or anyone should spend a megabyte for quality.

  Rather I was trying to give an example that spending lots
  (or what seems like lots) of resources on testing, logging,
  and other quality-specific code might be a good economic
  trade-off.  I chose a megabyte because I thought it would
  both seem wantonly wasteful to people, while actually only
  costing $3 to $10 per system.

++++ New Post, Same Topic ++++

From: John Cameron 
Subject: Re: OS Testing

>In order for the testing to have maximum validity, it must
>run on the version of the OS we ship, rather than an
>instrumented version with hooks for special testing. This
>means that our testing relies on the execution of tests in
>user space, which mimic aspects of the the applications
>normally run on our system.

The embedded stuff we ship is the instrumented version so
there is no difference between the code we debug and
the code that drives our machines.  That has to be an
advantage in assuring correctness.

>The above problems tend to be load sensitive and
>intermittent in nature. So, I run different mixes of the
>small test applications to generate a high load on the
>system. That tends to flush out problems with interrupts
>handling in the devices that are tested, as well as other

Are your test repeatable?  If not how do you tell if an
error occurred?  Seems like it would be very easy to get
a bad location on a disk or to mismap memory without
noticing or to fire off some process at the wrong priority
without noticing.  Are you relying on the results of the
applications as your only feedback?   Hate to be a pest,
but am always looking for new ideas.

++++ New Post, New Topic ++++

From: John McGrail 
Subject: Re: attempts at controversial statements


>  John is technically correct, but I wonder what you recommend
>  instead?

I follow the approach of "write it all down".  I write down
all the tests I can think of.  Even the odd/undoable ones and
ideas other people come up with to.  Then as I automate them,
I mark them done.  If a test can't/won't be automated, I write
the justification in the plan and mark it NYI (not yet

This serves two purposes:

1) Six months later I (and others) know why a particular test
   wasn't implemented.  No head scratching or past email
   surfing to figure out why.

2) Since I consider defect-free software a lofty goal, but an
   unrealistic one, I think it is as important to write down
   what was not tested (and why) as it is to write down what
   will be/is tested.

This gives others a chance to review & comment on my
decisions. And it prevents anyone from coming back later on
and saying "Why didn't you test X?".  If they ask, I answer
"it was on the plan that you approved and the reason for not
doing was ....".  In practice, the people I've worked with
don't do this (unless they are outside the review/approval

>  I find this attitude (which John may not have) very common
>  among QA people and some programmers.  They concentrate on
>  the theoretical limit, which in this kind of argument can
>  be shown to be too expensive in some dimension.  The
>  argument seems to be "If it's theoretically impossible,
>  we should reject it as a goal."

I definitely don't have that attitude.  While I'm a "can do,
but it'll cost X" kind of person, I've encountered many people
who are "can't do" people - even when you are asking for
something realistic.

If you ask me to do something that's wildly unrealistic, I'll
try to come up with numbers for how long it'll take or how
much it'll cost.  That's usually enough to get the person to
back down.  This length/cost info is also included in the plan
so it can be reviewed and revised later on.

My feeling is "... we should reject it as a goal __today__."
The reason for the added word is that time and business
pressures may change tomorrow and the goal may become
achievable either technically or resource wise.  Therefore any
goal you've rejected should be written down so you can
re-evaluate why it was rejected when things change.

John McGrail
American Internet Corporation -
1 (781) 276-4568

++++ Moderator Comment ++++

  First, let me apologize for even unintentionally sullying
  your reputation!  It's obvious that you have a lot to offer!

  Second, let me thank you for your thoughtful posts.

  Third, let me re-phrase my question:  Why are you so opposed
  to David Bennett's use of the word "MUST" in stating his
  software-quality goals/techniques?

  What harm comes from striving for them?

===== NEW POST(S) =====

++++ New Post, New Topic ++++

From: Travis Siegel 
Subject: self diagnostics.

I've been reading this digest from issue #2 (missed the first
one :)) And what I've seen here is mostly helpful and good
things, but what disturbs me the most is what I call the
Microsoft mentality in programming.  (No, I'm not bashing
Microsoft, it's just a name for the thing) Anyway, what seems
to be the current trend in programming is not to worry about
how large the program is, or how long it takes to do it's job,
just so long as it gets done.  One post in issue #11 mentioned
that an extra megabyte was no big deal, and that memory and
CPUs will keep up (and even surpass) these requirements.  This
specifically is what bothers me about the majority of
programming I've seen done in the past few years.  I'm a
freelance programmer, and although I still write exclusively
for dos (yes, there's still people using it) I don't believe
my assertions are any less valid simply because I'm not in a
windows environment. In any case, my point is, that After I
have a product ready (I think) for delivery, I give it to a
few friends, and even try myself, to run the program, and
actually try to break it.  Assuming that they don't find any
bugs, I then do my best to keep the program as is, but trim
down the code in the program to make it smaller and faster.  I
personally like the phrase, "First make it work.  Then make it
work fast." I don't know who said it first, but it's the way I
handle all of my programming tasks.  I first write the program
to get it working.  I then go over each routine with a fine
toothed comb, to see if it can be made more efficient, or
slightly faster.  I understand that in larger environments,
this isn't always possible, but the general attitude of
"there's plenty of space, and it's cheap to get more anyway"
or "Memory is cheap, they can just add another 16MB" are to me
sacrilege.  I honestly don't understand this practice.  Don't
people *want* their programs to run faster and better? Why
rely on upgrades of hardware to make this happen? Why not make
the program do the best job it can from the start? I'd be very
much interested in hearing other's opinions on this topic,
though I'd certainly not intend for this to become a flame war
of any kind.  I'm really curious on what others believe in
this regard.  Thanks much.

++++ Moderator Comment ++++

  Since I am responsible for the megabyte comment, let me

  I may never live down my "It's only a megabyte comment"!

  As a defense, I would say that I don't believe or recommend
  that you or anyone should spend a megabyte for quality, just
  because you can.  I do believe and recommend that you should
  spend an economically justifiable amount on code to aid
  quality, and that could be

  Rather I was trying to give an example that spending lots
  (or what seems like lots) of resources on testing, logging,
  and other quality-specific code might be a good economic
  trade-off.  I chose a megabyte because I thought it would
  both seem wantonly wasteful to people, while actually only
  costing $3 to $10 per system.

  Actually, I just looked on the web at my favorite sites,
  and the price range was $1.50 to $4 per megabyte for standard
  SIMMs.  At these prices, it seems like you could afford
  almost any amount of quality code...  But wait, maybe it's
  disk space?  Here the prices are $.05 to $.10 per megabyte!

  My point is, we too often use the "cost of quality" as an
  excuse for not following effective practices -- even though
  the economics are strongly positive for following the
  effective practices!

++++ New Post, New Topic ++++

From: John McGrail 
Subject: Re: attempts at controversial statements

Hi Terri,

Sorry I sent the last posting to you instead of
software-quality. I did a reply to the message and forgot to
change the "To:".

Speaking of that ...  It won't happen again if you set up the
reply-to field in the digests to be software-quality... ;-)


John McGrail
American Internet Corporation -
1 (781) 276-4568

++++ Moderator Comment ++++

  1) My name is spelled with a 'Y', thank you.

  2) Okay, excellent idea.  I tried to do it with the software
    I am currently using (Eudora Pro 3.0, NTMail and NTList).
    See if this version works any better.

++++ New Post, New Topic ++++

From: Rita Williams 
Subject: What books have already been reviewed?

Would I have to look at all previous issues?

Rita Williams,Instructor     
Business & Computer Studies     phone:  250.365.7292x310
Selkirk College                Box 1200, Castlegar, BC V1N 3J1

++++ Moderator Comment ++++

  Right now, there are only three books reviewed... :(

  And you would have to look at all the issues, although
  that isn't so hard to either retrieve them, or to go to our
  web site and read them.

  However, I will be adding a page on our Software-Quality
  section of our web site, so that all of the reviews are
  easily accessible.

==== BOOK REVIEW ====

  (Let's lynch that moderator guy!
  Where is our book review!?  ;-)

The Software-Quality Digest is edited by:
Terry Colligan, Moderator.

And published by:
Tenberry Software, Inc.     

Information about this list is at our web site,
maintained by Tenberry Software:

To post a message ==>

To subscribe ==>

To unsubscribe ==>

Suggestions and comments ==>

=============  End of Software-Quality Digest ===============

[Tenberry] * [Software-Quality] * [Zero Defects] * [Automated QA] * [Consulting] * [Site Map]
Last modified 1998.4.7. Your questions, comments, and feedback are welcome.