No one in their right mind would say that white-box testing is the only way to test an application. However, in too many instances, black-box testing is, in fact, the only testing approach QA is empowered to use. Rightness of mind notwithstanding, I would argue that this is a mistake and that quality assurance engineers should always be executing white-box tests to assess product quality.
A Few Definitions
Before I go any further, let’s be sure we know what black-box and white-box testing is. The International Software Testing Qualifications Board (ISTQB) plainly states that black-box testing is done “without reference to the internal structure” while white-box testing, is “based on an analysis of the internal structure”. In short, white-box testing is meant to verify that, code is running appropriately while black-box testing ignores code and measures results through the application’s user-interface (UI).
Therefore, one could not argue that white-box testing is somehow superior to black-box testing. Each one is intended for a different purpose. My argument is that all software should be tested from the standpoint of the white-box, even when tests are executed through the UI.
An Illustration: Style and Size
To illustrate, consider the following example. Your team has been asked to create an online store application designed to enable customers to order t-shirts of different styles and sizes. There will be two styles, men’s and women’s. But each style will only come in certain sizes. They will be offering men’s shirts in sizes S, M, L, XL and XXL while women’s shirts will only have sizes S, M, L, and XL.
The Black-Box Approach
The typical black-box testing process will begin with the requirements being handed down to QA some time before the product is about to be released. Even in an agile development environment, QA may not have a clear understanding of what has been implemented until requirements or user-stories are given to them. Hopefully, the requirements will accurately describe how the style and size controls are interdependent, and the QA engineer will dutifully write a test plan that verifies the proper behavior. When he tests it, assuming all else has gone well, he will find it working, report that the functionality has passed his test, and move on to other tests. In short, he will have done everything he was empowered to do to ensure quality in this situation.
Let’s Dig Deeper
While this fictitious, but certainly realistic, situation seems benign, even commonplace, it describes a testing process that falls far short of what true quality assurance could be. The problem is that even though the QA engineer is in a unique position to perform a thorough evaluation of the product’s quality, his test inherently lacks depth. He has only answered one question about the application, namely “does this application do what the requirements document describes?” However, we know that this is not the only question concerning quality in software. There is also the question of what is the best we can do for the users of this product given the time and resources we have.
A Better Way
There is another, more insightful approach to software testing and it does not have to add unnecessary time or resources to the project. Consider the possibility of having tests run by someone who had been a key part of the development team that created it. I don’t mean that they merely attended the development meetings. I am talking about someone who was brought on during the first meeting with product owners. This is someone who was part of the discussion that lead to the requirements being written for this page. Perhaps she was in on the design meetings around the creation of this page. Perhaps she heard from the customer herself about what they would like users to be able to do with this page. While she doesn’t write the code, she has been given access to the code and has a high-level understanding of how it functions.
Like the previous tester, she may report this test case as “passed”. But, she may also report her other findings and communicate with the development team that the quality of this page can be further enhanced at the code level. This QA Engineer is a white-box tester, personally invested in the project, performing black-box tests with a deeper sense of what the term “quality assurance” means — to the product and its stakeholders.
Box, Box, Gray-Box
While the ISTQB does not seem to have a term for this type of testing, others have identified it as “gray-box” testing, and it represents a more comprehensive vision for quality assurance testing. Gray-box testing is a strategy that requires the involvement of one or more quality assurance team members from a projects inception. Making attention to product quality a key goal from the very start of development. Now, I may need to walk back my claim that white-box testing is the only valid testing procedure. Obviously, only testing the code using the white-box method leaves out a very useful black-box approach that verifies that users will have the quality experience the product was designed to provide. However, these need not be mutually exclusive approaches to testing. By taking this one example where gray-box testing excels in verifying product quality, I have shown that I am, in fact, of right mind, and that combining the two approaches is essential to true quality assurance.