White-Box Testing: The Only Way To Do QA?

Software development and debugging concept. Bug found in binary code with magnifying glass.

No one in their right mind would say that white-box testing is the only way to test an application. However, in too many instances, black-box testing is, in fact, the only testing approach QA is empowered to use. Rightness of mind notwithstanding, I would argue that this is a mistake and that quality assurance engineers should always be executing white-box tests to assess product quality.

A Few Definitions

Before I go any further, let’s be sure we know what black-box and white-box testing is.  The International Software Testing Qualifications Board (ISTQB) plainly states that black-box testing is done “without reference to the internal structure” while white-box testing, is “based on an analysis of the internal structure”. In short, white-box testing is meant to verify that, code is running appropriately while black-box testing ignores code and measures results through the application’s user-interface (UI).

Therefore, one could not argue that white-box testing is somehow superior to black-box testing. Each one is intended for a different purpose. My argument is that all software should be tested from the standpoint of the white-box, even when tests are executed through the UI.

An Illustration: Style and Size

To illustrate, consider the following example. Your team has been asked to create an online store application designed to enable customers to order t-shirts of different styles and sizes. There will be two styles, men’s and women’s. But each style will only come in certain sizes. They will be offering men’s shirts in sizes S, M, L, XL and XXL while women’s shirts will only have sizes S, M, L, and XL.

Your team creates a page in the application with two drop-downs, labeled “style” and “size”. When “women” is selected for style, the size drop-down will only allow users to select S, M, L and XL. When “men” is selected, all possible sizes can be selected, including “XXL”. Assume that to make this functionality work a developer created one or more JavaScript functions to “listen” for changes to the style control and hide or disable the XXL option only when “women’s” is selected. To make this illustration as realistic as possible, you can assume that this was just one bit of JavaScript in a complex script, and it was completed, as so much code is, under the pressure of time and competing priorities.

The Black-Box Approach

The typical black-box testing process will begin with the requirements being handed down to QA some time before the product is about to be released. Even in an agile development environment, QA may not have a clear understanding of what has been implemented until requirements or user-stories are given to them. Hopefully, the requirements will accurately describe how the style and size controls are interdependent, and the QA engineer will dutifully write a test plan that verifies the proper behavior. When he tests it, assuming all else has gone well, he will find it working, report that the functionality has passed his test, and move on to other tests. In short, he will have done everything he was empowered to do to ensure quality in this situation.

Let’s Dig Deeper

While this fictitious, but certainly realistic, situation seems benign, even commonplace, it describes a testing process that falls far short of what true quality assurance could be. The problem is that even though the QA engineer is in a unique position to perform a thorough evaluation of the product’s quality, his test inherently lacks depth. He has only answered one question about the application, namely “does this application do what the requirements document describes?” However, we know that this is not the only question concerning quality in software. There is also the question of what is the best we can do for the users of this product given the time and resources we have.

A Better Way

There is another, more insightful approach to software testing and it does not have to add unnecessary time or resources to the project. Consider the possibility of having tests run by someone who had been a key part of the development team that created it. I don’t mean that they merely attended the development meetings. I am talking about someone who was brought on during the first meeting with product owners. This is someone who was part of the discussion that lead to the requirements being written for this page. Perhaps she was in on the design meetings around the creation of this page. Perhaps she heard from the customer herself about what they would like users to be able to do with this page. While she doesn’t write the code, she has been given access to the code and has a high-level understanding of how it functions.

What if when this tester begins QA work on the page, she does so from an entirely different perspective? Like the previous tester, she will create a test plan that aligns with requirements, testing the size and style drop-downs appropriately. However, perhaps she recalls that the customer mentioned wanting to add more styles later, such as girl’s or boy’s. Along with that, she’s had a look at the JavaScript that runs the page and she would like to open a discussion with development about implementing this behavior in a way that makes expanding or altering the options list easier and less error-prone in the future without introducing a lot of risk to the project at hand.

Like the previous tester, she may report this test case as “passed”. But, she may also report her other findings and communicate with the development team that the quality of this page can be further enhanced at the code level. This QA Engineer is a white-box tester, personally invested in the project, performing black-box tests with a deeper sense of what the term “quality assurance” means — to the product and its stakeholders.

Box, Box, Gray-Box

While the ISTQB does not seem to have a term for this type of testing, others have identified it as “gray-box” testing, and it represents a more comprehensive vision for quality assurance testing. Gray-box testing is a strategy that requires the involvement of one or more quality assurance team members from a projects inception. Making attention to product quality a key goal from the very start of development. Now, I may need to walk back my claim that white-box testing is the only valid testing procedure. Obviously, only testing the code using the white-box method leaves out a very useful black-box approach that verifies that users will have the quality experience the product was designed to provide. However, these need not be mutually exclusive approaches to testing. By taking this one example where gray-box testing excels in verifying product quality, I have shown that I am, in fact, of right mind, and that combining the two approaches is essential to true quality assurance.

About Aaron Priest

Aaron PriestFor as long as I can remember, I have been curious about how things work. In fact, it was a curiosity about the software I was dealing with every day that lead me from a career in education to learning how software and its related mechanisms work. As a Quality Assurance Test Engineer for Saturn Systems, analyzing software, both where it works and where is doesn’t, is a part of my everyday experience. However, I was surprised to discover that the most fascinating aspect of software development is the human beings that make it. How people work individually and how they work in teams to accomplish a goal can be as intriguing as the latest technology. I enjoy discussing not only software, its nuances and pitfalls, but also the journey people take from simple ideas to making useful creations.

2 comments on “White-Box Testing: The Only Way To Do QA?

  1. Avatar
    Kurt Johnson on

    Aaron, I have a few questions about an upcoming initiative I am thinking quite seriously about that encompass the information in this article. I am interested in developing software that performs a particular task, but doing it in a way that provides the user interface the functionality the user wants, but also in a way that is 1. friendly to modifications the coder may have to make in the future, 2. functional in the greater code-set with respect to performance and 3. doesn’t affect the performance of other, related code-sets.

    I feel somewhat uneducated in this respect and wonder if you can point me in the direction of reference material that would open this area up for me a little. I need to think more of code, not just to solve a problem, but to do it in a symbiosis with the code that is already there. In other words the whole platform should be designed from the ground up so internal processes help each other, not hinder performance or make it harder to modify in the future. Wow, I hope I am making myself clear.

    • Aaron Priest
      Aaron Priest on

      @Kurt Johnson, you are coming through loud and clear! As a quality assurance engineer, I am happy to hear that you’ve adopted a philosophy of creating today’s code with the future in mind, knowing that future changes and rewrites are both inevitable and potentially risky. I should note that while I am not a software developer per se, I’ve seen enough to know that what you are probably looking to do is “modularize” your project.

      There may not be much that I can suggest right now in terms of resources because what you will end up doing will depend on the technology and language you will use as well as the type of application you intend to create. But I’d be willing to bet that there is a host of materials out there on the topic of modularization for your specific situation.


Leave a Reply

Your email address will not be published.