The Tester’s Dilemma

Even though at heart, both software developer and software tester have the same goal, which is delivering a software project with good quality, they go about it with completely different mentalities. The software developer has a constructive mentality, he/she is the builder, they think about creating things from other things that may be similar or may have remotely the same features, or even create things completely from scratch. While they’re working, they’re thinking “I want this to work”. They think about the business from a number of points of view and start tackling these visions they have of how the system should work. Once the system is working in a way that satisfies their idea of “working”, they have achieved from 80 to 100% of their goal and the feelings of satisfaction and achievement start to kick in.

The software tester, on the other hand, has a destructive mentality, but not in an evil way. The tester starts working on the project by doing a thorough analysis of all the things the system should and should not do. They map the SUT to the requirements in a way that “should” ensure the system does not misbehave so to speak. At the next stage of testing, the destructive mentality kicks in. the software tester starts working with the purpose of finding the bugs in the system; trying to find every possible mistake, misbehavior, and anomaly, however improbable it may be. They think of all scenarios, in an attempt to capture any idea the customer may have had about the behavior of the system that was not conveyed clearly in the system’s documents and may, by chance even, occur after delivery. And so we test and test and test, we look for bugs and misunderstandings, only wishing we find all the most probable bugs because we know that we will never find them all.

Here lies the dilemma of the software tester. We execute test cases upon test cases with limited time and budget. We usually do priority based work, where we execute the most critical test cases from the business point of view then we move on to the improbable scenarios and tinier bugs. Yes, we get our satisfaction and sense of achievement from finding other people’s mistakes. The problem is we will never be fully satisfied. The developer may feel at peace after fixing most of the bugs in the system, but the tester will always know that there are still more bugs that may come up. The tester knows that humans make mistakes and even though it is their job to find other people’s mistakes, they know that no one will find their mistakes unless it’s too late since the system will most probably be delivered to the customer live at this point. At this stage, the impact of the tester’s mistake will be too great to be “just a human mistake”.

Ok fine, I’m being a bit too morbid. In my case as a software tester, I can never feel satisfied when the system is delivered, no matter how much effort and time I put into the work. I always have the feeling that I’ve missed something, or that I haven’t worked hard enough. I’m always worried about the what-ifs. At the end of the day, I know the system I delivered isn’t perfect, and I know it is buggy even though the bugs are probably beyond noticing. But that knowledge will always take away my sense of satisfaction and achievement about a job well done.

Technorati Tags: software,developer,tester,mentality,SUT,achievement

Reduce, Reuse, and Pairwise!

I’ve come some projects where repeated components seemed to be the trait that stood out. This was by far every automation engineer’s dream; to write some code that could be reusable across lots of different test cases. This however has proven that it could be every automation engineer’s nightmare too.

To be able to have a chunk of code with a size fits all character; it has to be quite configurable. In order for this code to be configurable, there are going to be parameters, flags, counters, and quite a bit of control words. For anyone who studied some logic design, this code could be used nicely as a multiplexer of a sort, with different control words giving different results. The problem with this is that getting lost in all the configurations could turn into a nightmare, especially if it is used across different test cases; one test case could require adding 5 items to a grid for example, another would require removing these items, a third may not even relate to the grid in the first place. These could be controlled by 2 control words for example: items added, items removed, if both are nulls then we can just leave that grid alone. But what happens if there are other control words related? What happens if this is not the only reusable chunk of code? MAYHEM!

The best way to handle this is through proper design and figuring out exactly what is needed to be done in this module, it would be very hard to iterate with new and different requirements, so a bigger picture of the whole module needs to be figured out from the start. The features, however, need not be scripted at the start as soon as they’re all catered for so that later on no messes take place.

The other trap that may come up with the reusable code issue is simply GREED! Once you have some code that can be used with lots of scenarios, you will want to run each and every scenario this code is capable of. So let’s say you have 5 control words for one chunk of code, assume they’re all Boolean values to make things easier. In order to get all the possible scenarios – do not tell me it is not tempting to think about exhaustive testing – you will need to have 32 automated test cases. You will end up with more test cases that you actually need, which is weird because us testers are always pressed for time and the more time you give us the more tests we’ll run, but what good does it do in acceptance or regression tests to cover the testing exhaustively. It will take just as much time to analyze and maintain. In addition, having isolated parameters will not really help all that much with finding new bugs, so it would be a waste of time.

Luckily, I came across this really cute, smart, and effective idea about handling testing in a non-exhaustive manner while still maintaining a nice % of coverage; Pairwise testing! For those of you who haven’t heard of pairwise and y’all call yourselves testers, shame on you and (for you who know Arabic, thakalatkom ommokom) go to http://www.pairwise.org/ for more info on the matter. The whole idea of pairwise is that instead of having an exhaustive set of tests (the 32 we talked about before), any 2 parameters can be used to generate a failure. So you need to cover all the values of parameters in pairs instead of exhaustively. We’re engineers so we won’t talk about how the algorithm works. How does this relate to reuse in automation? That’s a secret :P

Okay not a secret, what if we use pairwise to generate the different sets of control words, that way, we can have a smaller number of test cases with a decent degree of coverage, and with larger sets of control words, this would be quite the charmer. There are a lot of tools that could generate pairwise combinations of certain parameters, thus giving us the amount of test cases we want without wasting time in test runs and analysis. Pretty cool huh ;)

Code reuse in automation can be a double edged sowrd that needs to be used wisely. Having less lines of code could seriously compromise its complexity which will cause a disaster in script maintenance that could ruin the entire test suite with a tiny bug (yes automated code has bugs… it’s CODE!) so I am stressing – AGAIN – on design and objective before going into automating scripts. And God help us all!

Your mission – should you choose to accept it –is to automate! –Part 2

No, I'm not done with the objectives of automation :)

My purpose is not to cover the objectives of automation as much as it is to cover the idea behind the objectives we define. Testing, and test automation, is like anything else in the SW process. It needs design, going into details, and having a clear idea of what the heck is going on. Testing may cover that, but test automation does not always have that cleared out.

So I discussed my idea of how to automate tests for regression purposes, and I believe that the mission was accomplished with maximum return on investment and minimum extra work. But in the case of acceptance testing, things may get a bit tricky.

Again, before laying upon you my great ideas (kiddiiiing :)), we'll have to define the objective of acceptance testing. By definition, acceptance testing is to make sure the system under test conforms exactly to the customer requirements. Acceptance testing is not done for the purpose of finding bugs in the system. So the objective is to make sure every single detail the client has agreed upon to be a system requirement is covered in the acceptance test and is present clearly in the system. It is also to make sure the system behaves "Expectedly"; i.e. there are no crashes and no surprises due to deployment issues or different environments. So the objective of acceptance testing will be split to 2 parts: firstly, there will be the regression test objective mentioned before which is covering all the bases; no crashes, the scenarios are completed till the end and committed, etc. And secondly, the automated script must map the requirements to the T. so if the requirements say that a password field for example must not be less than 6 characters, then an automated acceptance test case is to make sure the password text field does not in fact take less than 6 characters.

Come to think of it, if the automated acceptance test cases cover the requirements to the T, wouldn't that be too much work? I mean, what were we doing during integration testing then? We already tested all the requirements and then some. We confirmed that everything that is "code related" is working as good as expected and we decided to release so the quality must be somewhat good! This is in addition to all the negative testing of course, that goes without saying. So why cover every single detail if I, as a tester, am confident about 70% of the details? So does that mean we can alter the testing objective?

Of course we can, but we won’t! let’s consider the manual testing perspective. Manual acceptance testing covers the most critical forward scenarios, the most critical business rules, and if there is still time, manual testers go a bit deeper into the less critical scenarios and rules, and maybe, they cover some commonly used negative scenarios. If we map this testing to automation testing then we will have to follow the same concept; first, we cover the basic scenarios of everything (if there was a regression automation suite ready then that would be sufficient), then we cover the basic business rules which may or may not be covered in the regression automation suite. Next, we start covering the less critical forward scenarios. This shouldn’t take much time because there will be reuse of the scenarios already covered earlier, same issue with the less critical business rules. Finally, if there is still time, some negative scenarios should be covered just to put your conscience to rest.

Time wise, this shouldn’t take that much time if there is already a core automation suite ready. But this will not be the case most likely. The process of adding iteratively to the testing suite may cause a lot of problems in the maintenance and use of the suite. In addition, tracking bugs would be a pain. However, the most critical problem would be maintaining a stable version. It’s starting to sound a bit like the problems the developers face. That’s because they are the problems the developers face. Will adding a new feature to the test suite break another feature? Will the tracing of the code become impossible? Will we end up changing core functionalities in the code to accommodate the changes? Most probably yes! So when is automation good for acceptance testing? When there is proper design of the test suite, but that’s a story for another day.

Let’s assume the happy scenario when the test suite was properly designed and planned for. Does the automated suite help now? Of course it does, with maximum return on investment. Consider manual testing again, if it is done systematically, testers rarely have the time to cover all the scenarios and rules mentioned. In fact, they barely cover half of them. Manual testers rely on their intuition in acceptance testing, based on their experience in testing the system. The automated test suite does not have gut feelings, but it is much faster. This speed allows for an automated suite to cover more test cases in acceptance testing, thus increasing the return on investment of the acceptance testing. If we assume that the time taken to automated the acceptance test suite (and by that I mean the additions to the suite not the core) and the time taken to run and analyze the automates tests, is the same as the time taken to manually do acceptance testing, the ROI will be higher than that of manual testing because the coverage will be higher, thus giving more confidence to the team and customer. Mission accomplished! (Again)

Your mission – should you choose to accept it –is to automate! –Part 1

The automation is strong with this one…

Writing automated test scripts is a very important task that we perform from time to time on almost all projects. Why do we write automated scripts? Because we'll do the regression through them, because the acceptance testing will be completely automated, because we have to??!!!

Don't get me wrong, I'm not making fun of automation or its value. In fact, I am a HE-UGE fan. I'm just wondering, are we doing the automation for the right reasons using the right techniques?

I've worked on the automation of several projects. And it was always an easy fun task… well, not always easy to be honest, not always fun either. Okay, okay, I liked it when it was fun and a little complex, and hated it when it got repetitive and boring. But that is not the point! The point is that there was always something to automate. Make sure the scenario "works". Now doesn’t that statement sound a little familiar? Didn’t project managers and development leads always tell us that 5 letter statement whenever we were testing a release with critical time? And didn’t we always tell them that this is not how testing works? It's a shame to find those exact words coming out of the mouth of a tester. In fact, it is a shame to have the exact opposite come out of the mouth of a tester: "your scripts must make sure everything works!"

Just because automation may seem like the light at the end of the tunnel doesn’t mean that the end of the tunnel is Utopia. We have to know why we are writing automated scripts exactly, how long it would take to analyze the results. Hell, we have to know what we need to have in the test results in the first place. But above all, we must define the objective of the automation mission.

I've mentioned earlier two broad ideas of objectives for the automated tests. One is regression testing and the other is acceptance testing. Let's start with the first one:

Regression testing is the task of making sure the fixes for old bugs has not introduced new bugs. The action itself is broad level testing; a quick test cycle across as much of the system as possible, so the objective of regression testing is to cover all the major bases of the SUT. That means that the objective of the automated scripts for the SUT is to cover all the bases (same objective as the regression test); to make sure no crashes take place, all the scenarios are completed and committed, the major items of every page do in fact exist and are working. The details of the above few statements depend on the system and its maturity level, the definition of major highlights in the system, and the definition of the most critical scenarios. But the main objective has been defined: Major System Highlights. Under an agile process, the team may have a release weekly. That means the average tester will have to do regression on the old part of the system, test the new functionality, and automate the new functionality for next week's regression ever single week. Now if we consider a complex of big release where a tester is required to automate "everything"… say 50 test cases for example with the option to test several modules with several hundred data inputs to make sure that there are no bugs as a result of data, that could take them a month to code, a week to maintain with every release, and another week to analyze the results of the automation. This will make testing using automation a huge bottleneck! At this point, the ROI of the automated scripts will be in the red. In addition, the tester will most probably decide to stick to manual regression (fast regression while only looking at the highlights of the system) that to waste time with extensive automation.

On the other hand, if the automation task had the same objective as the actual function it is supposed to do, which is regression, then preparing the test scripts and the data for a week long moderately complex release would be 12 to 16 hours (just an educated guess here). So if the release took 40 hours to develop, and the actual testing took 40% of that time, the testing effort will take 16 hours, scripting will take another 16, regression on the old release would take 4 hours, and the maintenance will take another 4 hours, then we're talking about a total 40 hours. Voila! Mission accomplished J

Getting the ideas out of the freezer

I decided I wouldn't start this blog unless I have a couple of items written down first. I get these sudden bursts of energy where I can do a million things at once. But on the other hand, I can get sudden bursts of anti-energy when literally lifting a finger would be a tedious job.

So here I am, with a burst of energy (or maybe a sugar rush having had a snickers tart last night) willing to philosophize all about testing. I hope all of you virtual people who would read my blog would enjoy reading it as much as I am sure I will enjoy writing it… and I will definitely enjoy some serious (or not so serious) discussions :)