Beware the “Midas Touch”

by garyg 7. November 2010 23:36

So we are in the final go/no-go hour for a product release, and its not looking good.  My team had been charged with testing the product for last 3 day.  The development team was busy playing wack-a-mole with several P1 and P2 bugs that continued to regress.  Our QA team had no involvement at the beginning of this cycle, nor visibility to any hard requirements (I know, not a good start).  A particularly nasty P1 that appears to be in the framework of the product, is on its 4th try through the regression testing cycle.  This is one that the developer was sure he had it “this” time.  My confidence of course was not there, and I recommended this release be pulled.

The client of course was not happy and called an emergency meeting to get a handle on what was happening.  We were now going over the exact contents of what was supposed to be in this latest release, feature by feature.  Seeing the impending revelation creeping upon him the developer finally revealed there may be something “a little extra” in this code.  Something completely off the requirements list, project plan, and of course did not go through the fairly rigorous code review this client normally does.  We were unwitting victims of a classic Gold Plating gone wrong situation and nearly done in by the Midas Touch of unchecked development without matching requirements.  No malice of intent but the results were the same.

How this situation was resolved isn’t important, but how we could have prevented it is.  A simple, solid, Requirements Traceability Matrix combined with some training on early recognition of Gold Plating signs was on my list of recommendations.  I’ve often seen the costs of Gold Plating represented by actual extra work and schedule overruns.  This was the clearest case I’ve seen to date were it directly caused a quality issue of this magnitude.  A good lesson for us all.

5 steps to “better” bugs

by garyg 11. June 2010 11:07

While I’m sure we’d all rather avoid defects in our software all together, in a sufficiently complex application bugs (or regressions as we also call them) are as inevitable as death and taxes.  As an old mentor used to tell me, if you aren’t failing some of the time, you aren’t trying hard enough. The trick of course is catching them before the risk they present is fully realized, i.e. in front of the end user in a production environment.  Finding and remediating bugs falls under Project Quality Management in PMBOK (Project Management Body of Knowledge, Chapter 8).

So as long as you have them, you might as well get your moneys worth out of them.  Here’s how.  A “bug” in your software system represents a defect that needs to be remediated in some way, as effectively as you can, and without breaking anything else in the system.  So how do we achieve that ?  Well we need to find them early first with good testing in a well managed development cycle, but that's the subject of another future post.  Once we have located them we need to properly assess them.  At a minimum you want to know:

  1. What the defect is and what it impacts in the system.  This begins with a thorough description of the problem observed.  We are looking for a well written observation here, more than just a “its broken” but shorter than your system specification.  A paragraph should be about right.
  2. How do you reproduce it.  If the developer cant reproduce it, likely it will get closed as a “non reproducible” bug and will be a complete waste of time for everyone involved.  Be as exact as possible with clear steps and any particular data needed.
  3. Screen Shot or Video.  Window 7 has the built in Snipping Tool, but there are a lot of them out there.  The idea is a visible representation of what you just described.  Full motion video is even better for complicated issues. Microsoft has Expression Encoder 4 available for free , but there are many out there.
  4. Priority and Impact.   I’m a big fan of two level assessment of defects.  The first level, Priority should indicate how bad the problem on a numerical scale.  P1 meaning there is no way you can go forward without releasing it, down to P4, an inconvenience.  Impact should indicate what the financial or effort based impact of this error is (I-1 to I-4).  Will it take 5 weeks to fix or do we just need to comment out a line.  Often the original tester will not be able to assess the Impact, but this should be done by the person triaging the defects.  In a project management capacity this really represents Qualitative and Quantitative Risk Assessment.  More on this two level system concept in a future post.
  5. How did we fix it.  This is a critical step.  After a resolution is developed and verified, you need to detail it right in the defect document.  While we don’t need the source code here, you should link or reference the change set that fixed it.  Also, always be on the lookout for a best practice or new design pattern that worked well in approaching this defect or one that didn’t.  This is about building up your knowledgebase.

One last parting thought on defects.  While there are hundreds of theories and best practices on Software Quality Assurance and reducing bugs (and certainly worthwhile).  I like to remind people of a famous quote from Thomas Edison on the thousands of failed attempts to create the light bulb: “I have not failed. I've just found 10,000 ways that won't work.”  As long as we keep that in mind and learn from the mistakes we make during the creation process, we can indeed extract value from them rather than just writing them off as just being a costly byproduct.

Preventing multiple plug-in request calls with IsRedirectFollow()

by garyg 10. April 2010 10:01

Figured I'd share something I found valuable.  Did you ever make a call to a request level plug-in in a Visual Studio 2008 WebTest and get multiple calls to the same plug-in because of a 302 redirect?  Well I did and took me a little bit to find out a way to prevent it.
When you are making the call in your code, decide if it should run based on IsRedirectFollow property (
As an example:

  1: namespace MyAppTests
  2: {
  3:     public class GetSomethingPlease: WebTestRequestPlugin
  4:     {
  6:         public override void PostRequest(object sender, PostRequestEventArgs e)
  7:         {
  8:             if (e.Request.IsRedirectFollow == false) //only want to run this on on a primary, not a redirect
  9:             {
 10:                 //do something here
 11:              }
 12:          }
 13:       }
 14: }

Anyway, I hope this helps someone else a little further along.  It works in VS2008 and VS2010 as well.  I'm sure there could be a more efficient way, but this worked in a pinch ;-)

Using Visual Studio 2008 Web Test Request Plug-In to Check results in a SQL DB

by garyg 10. April 2010 06:12

Some of my regular readers said they wanted to see more "technical" content, so here is one that perplexed me a while.

While assisting a client with setting up an automating testing environment using Visual Studio 2008 Web Tests (among other things) we uncovered a need to check the results of a transaction halfway through, then when its complete to verify the results of the test.

Now I know what experienced testers and SQA people are thinking, "why didn't you just use an Extraction rule from a results page and validate that?". Yes, that's the first thing I wanted to have done as well and that works quite well under most circumstances.

Unfortunately on this particular application there is no real "confirmation" screen that displays the results of the transaction, just kind of a yeah I did or no I didn't kind of page. Not good enough in our case where I wanted to have real transaction results.

Since time was very short and I'm still working with the group to adopt more "test friendly" designs we needed a sure way of verifying the results. I though that this check DB feature would be a native feature in the VSTS2008 Web Test, but the only DB connectivity included out of the box was binding to a DB for data driven testing (which is very useful as well).

So our option was a to create a request level plug-in to go out to the DB and check the transaction results, and write them back to the test results.

My goals here were:

  1. Connects to an SQL DB.
  2. Builds a query string using a Context parameter
  3. Puts the value pulled from the DB back into another Context parameter for use in follow on requests.

Here is how I did it (in a plug-in 101 type format), complete with code snippet I used to create this:

1. Create the following in a class library in your test project (that part is covered in detail in MSDN), compile, and reference it:

  1: using System.Text;
  3: using Microsoft.VisualStudio.TestTools.WebTesting;
  5: using System.Data.SqlClient;
  9: namespace Test1
 10: {
 12:     public class MyRequestPlugin : WebTestRequestPlugin
 13:     {
 15:         public override void PostRequest(object sender, PostRequestEventArgs e)
 16:         {
 18:             base.PostRequest(sender, e);
 20:             int CustomerID = 0;
 23:             // this is my connection string
 24:             String connectionString = "Persist Security Info=False;Initial Catalog=dbname;Data Source=machinename; User Id=dbuser;Password=somepassword";
 27:             // select statement getting just the field I need.  Note that if this is;
 28:             // messed up it may throw an error saying is cant open the db.;
 29:             // This is misleading, its probably your select; 
 30:             SqlConnection connection = new SqlConnection(connectionString);
 32:             string queryString = "Select CustomerID from Orders where OrderID=" + e.WebTest.Context["OrderID"];
 36:             SqlCommand command = new SqlCommand(queryString, connection);
 38:             command.Connection.Open();
 40:             SqlDataReader reader = command.ExecuteReader();
 42:             while (reader.Read())
 43:             {
 45:                 CustomerID = Convert.ToInt32(reader[0]);
 47:             }
 49:             e.WebTest.Context.Add("CustomerID", CustomerID);
 53:         }
 57:         public override void PreRequest(object sender, PreRequestEventArgs e)
 58:         {
 60:             base.PreRequest(sender, e);
 62:         }
 64:     }
 66: }

2. Insert the Request Plug-in (if you compiled and referenced it, it will be in the list.) AFTER the Context parameter you are using (in a production test you'll need error control, errors in a Plug-in are ugly and will mess up your results).

This whole exercise made think that a "data check" validation rule of sorts really should be part of this product.

Anyway hope this helps someone else a little further along some day using web tests. This same method also works in Visual Studio 2010 as well.

About the author

Gary Gauvin is a 20+ year Information Technologies industry leader, currently working as the Director of Application Lifecycle Management for CD-Adapco, a leading developer of CFD/CAE solutions. Working in both enterprise environments and small businesses, Gary enjoys bringing ROI to the organizations he works with through strategic management and getting hands-on wherever practical. Among other qualifications, Gary holds a Bachelor of Science in Information Technologies, an MBA, a PMP (Project Management Professional) certification, and PSM (Professional Scrum Master) certification.  Gary has also been recognized as a Microsoft Most Valuable Professional.

LinkedIn Profile:

(Note: Comments on this blog are moderated for content and relevancy)


Month List

Page List