Exploratory Testing - Positive and Negative points

Positive Points

• Has a planning structure of testing ( called charters)
• Find defects through exploring some pre-defined (planned) areas of testing
• Do not rely in any formal documentation to be created.
• So, it is useful when you do not have lots of documentation for each steps of the software or if this documentation changes often.
• Easy to maintain and change
• Minimize the time of test design (since it is design and execution at the same time)

Negative Points

• Highly depends on the test experience
• Easily misused
• Can need to have a complementary test scripted to assure test coverage.

Study Cases  

Ad Hoc - Experienced tester & Error guessing

Project scenario: Application to validate promotions from telecom operator. (E.g. Mammy’s day), Development team: Outsourcing, Test Team: Outsourcing (not same company from dev team). Both were working in same client, in his own house.

Solution: It had been applied Ad Hoc tests added to error guessing technique by getting input from Product Manager (from marketing area), who informed problems found in previously; Testers used past experience and creativity to determine the scenarios may cause errors.

Key point: Testers experienced, even they hadn’t experience to the application the testers had a lot experience in software testing.

Checklist - Sanity tests 

Project scenario: Four outsourcing companies working on same software where all leads from each one had a weekly meeting and test team had to report sanity test result.

Solution: Testers weren’t able to execute tests in details following steps in same day that the build was make available to be tested. So, it has been created a checklist where test points were raised and test team used to execute exploratory tests based on that checklist. The test type was sanity (basic), just to identify high priority bugs on new build.

Key Point: Experienced testers in software testing and strong skill on that application.

Session Based Testing & Charter

Project scenario: Agile (Scrum), 1 tester, 12 developers and 1 Scrum master – All team was on the same location. The client was located into a different city (same country).

Challenge: The tester as the whole team received the stories at the same time, and joined the meeting with the PO (client side) to better understand the client scenario. Discuss some technological solutions and raise blocks (scenarios that were not covered by the documentation that needed to be clarified). While the development team was working on the code creation, the tester was planning the chart (created the missions, areas covered)

Solution: The use of chart proved to be very useful since it didn't relied on detailed step by step and allowed a “free time” to focus on learning how to use the product. During the chart execution, it helped to keep the focus and concentrate on the vulnerability areas. Also, the test do not got blocked by not developed yet functionality's, once it didn't not contained detailed step or dependencies between the steps

Bugs were found, value was added and the client was satisfied with the software quality.

Key Point: Experienced tester in software testing and agile methodologies.

Conclusion & recommendations

In a conclusion we can affirm following:

Get escaped bugs: There are some studies that are concerned in a linear way to executing tests mainly in a scripted tests, the point is when these tests are executed several times and in a several version it might become vicious testing and didn’t found effective bugs the result is exploratory tests are able to found escaped bugs.

 Fits in several project scenarios: It is a good practice to performing tests in an agile or not agile projects, in additional it fits in a different test type, it means exploratory testing can be used as system, component, regression, integration tests and so on.

 Used according to the test strategy: It can used in additional to test strategy mainly on saving time.

  Improve test’s effective when used as additional method on test phase: It is better applied when it is used as additional execution method that does why will assure the test coverage independent of development phase or test type. If exploratory testing is applied as complement of these tests the chance to find different bugs are improved.

References:

[1] http://www.quardev.com/articles/slides_breaking_down_and_building_up_exploratory_testing_skill

[2].http://testobsessed.com/2006/04/19/rigorousexploratory-testing/RigorousExploratoryTesting

[3]. http://blog.platinumsolutions.com/node/35

[4] Oxford Advanced Learner's Dictionary of Current English, A S Hornby, OXFORD – Oxford University Press 2000

Exploratory test techniques

Ad Hoc

            Ad Hoc tests are executed without any plan or structure. Tester just open the application and going through, browsing according to his creative.  Bugs are found by accidentally and it is hard to reproduce, due to Ad Hoc doesn’t follow any structure. 

If you take a look only in the word "exploratory", it might be acceptable to say that the tester are performing a kind of exploratory testing, but defini tivelyit can not be classified as exploratory tests at all!! It happens because this test is performed without any plan, design or structure.

So, the question arises: Is there any scenario to apply Ad Hoc tests?

The answer is yes, there are!! It will depend of the strategy defined before testing. Following is some examples that this tests might be a good solution:

•  Software’s stability: During code freeze phase or as part of regression tests, Ad Hoc might be a good practice to help on get software’s stability due to task force and browsing in the software without plan or pre-requirement.
• Get knowledge about new software: When new person is joined to the team, in this case find a bug is "plus" of learning process.
• Hands on by leader/manager: Test leaders/Manager tend to don’t test software in details due others tasks, Ad Hoc test could be applied to get feeling about the software which he is responsible.

Also can be exist others situations to apply Ad Hoc tests, once more, it will depend of the strategy defined. Testing software is a creative area, these examples are just to give you some ideas to define the best strategy for your project.

Checklist as guidance

An checklist with highlights about software to be tested, is created before performing tests. It means only main function without details.

Let’s suppose that it is software to use SMS function from your mobile device. There are options to send, receive, delete, save as draft, based on these main functions, an checklist is created. Take a look below:

Checkpoint	Test result
Send SMS
Receive SMS
Delete SMS
Save as Draft

The checklist is used as guidance, during exploratory testing. The tester can also use his creativity to explore each situation, see below.

Checkpoint	Examples of scenarios to be explored
Send SMS	concatenated, not concatenated, in roaming, not roaming and so on
Receive SMS	connected to phone, not connected, during a calling or not and so on
Delete SMS	delete saved SMS, not saved, in inbox, sent box and so on
Save as draft	when receive a call, when leave the screen and so on

Note that is important, the setup and clean environment for each checklist line, to garantee the traceability and bug reproducible.

Session Based Testing & Charter

Session based testing was created by James and his brother Jonathan Bach as a need to track the testing they were performing. Since Exploratory testing is a king of a ad hoc processing, not restricted to pre-defined test steps or test procedures with mission of  finding bugs without previous noticed and find them fast sessions-based testing were created.

Session based test management provide a way for the testers to make orderly reports and organize their work without obstruction the flexibility that makes exploratory testing useful.

As definition of what is a session, sessions are time-boxes within which the testing occurs. Each session has a charter (a little mission) and results in a session report.

Basically, each test session (charter) can be composed of the following sections (but not restricted to):

• Mission statement: What you're going to test (Scenario)
• Areas to be tested: The functionalities from the mission
• Tester names
• Date and time used for the session. That can be a previous planning of a minimum time and a maximum time.
• Test Notes: Where the scripted part is created
• Issues: questions or problems that not necessary is a bug
• Bugs: The bugs found into the areas tested

Exploratory Testing - What really are exploratory tests?

Despite the fact that the concepts of Exploratory Testing and Ad Hoc testing are similar, and sometimes defined as: something, different names, we need to establish the concepts for each one of them. It is a very useful thing when you talk to someone about exploratory testing subject, for example, and ask: “What exploratory testing is for you?” From the person answer you got the idea and you can start on what exploratory is or isn't.

For this article proposes, we can establish the following:

• Exploratory Testing: were defined on the section above.
• Ad hoc testing: If we get the Oxford dictionary [*] definition: “arranged or happening when necessary and not planned in advance”. In other words, translating into the test word, it can be defined as: an unplanned in advance testing but used when necessary to solve a problem. Prepared for a particular situation, as James Bach's definition.

So, Ad hoc testing can differ from Exploratory by the lack of previous planning, but it does not mean that it is a worthless technique of testing. If fact, we can think as ad hoc as a part of the exploratory testing, once that you are creating your test as long you are running it.

We dare to establish that: all ad hoc testing are exploratory testing, but not all exploratory testing are ad hoc.

Despite of what may seem, Ad hoc are very useful in some situation or scenarios. Such as:

• Get to know the software: If you are going to a new project, or testing new software that you have never heard about, you might just what to navigate around without any scripts to know in what software you'll be working for.
  
  
• Verify the Software stability:  After all the bug fix phase on software, it is supposed to be ready to go. So, you might what to just navigate around, and see if everything is ok, or just take a last look into the software that you help to build.

Exploratory Testing - Introduction

Abstract

What comes to your mind when your heard the word “exploratory testing”?

Some can say that this is a way of test software in a short period of time, others may think that this is a friendly way to run tests, once that it do not follow a formal documentation, or even there are some that might think that this is just a “walk through” on the software functionalities and then assure the product quality.

Besides that, there are other questions related, such as: How can I apply Exploratory Testing on my software? Who should run this tests ? Exploratory Testing really assure the quality of the product ?  How can I report the results ?

This speech has as objective clarify this and other questions related to exploratory testing as well as present some case studies where exploratory testing were applied in real projects with different objectives and software’s.

Introduction

Not different than others aspect, exploratory testing is one more thing that is not strictly defined yet. That’s why I am writing it, to explain the suitable concept about exploratory testing and help you to understand; although this can also been improved and changed further. =)

Some managers tend to not trust on exploratory method due to lack of formal documentation and in other hand, there are some managers that think the “poking around” on application is a exploratory method giving enough tests to guarantee product’s quality; Actually it is neither one nor the other, exploratory tests is just different way to executing tests but it doesn’t means that it is a messy, it is just different way to find bugs and it is a pretty effective method to do it.

At first we have to realize that “exploratory” is very embracing word and it is also applied on technical side; It means that you can use “exploratory” way on performing tests like ad hoc or other one, you are really “exploring” the software and you are able to call it as a exploratory testing, but you can’t say that exploratory testing are ad hoc tests because they are completely different. What I am saying is: the ad hoc tests are kind of exploratory tests but exploratory testing are not ad hoc tests. It will be clear on further.

There also a formal technique to perform exploratory testing created by James Bach, when you can document and get the steps traceability, it is Session Based Testing.

This work explains what is exploratory testing, the difference among ET and Ad hoc, some techniques to perform ET, positive & negative points as well as some case study where has been applied exploratory tests using the same techniques detailed here.

Exploratory tests definitionS

I prepared some phrases found on some books/links to define exploratory tests:

Ø  “What’s the big deal? Exploratory testing is random pounding on the keys.  Nothing to it. My toddler does it every day.”[1]

Ø  … “Oh, Exploratory Testing,” said one of the developers, “that’s where the tester does a bunch of wacky, random stuff, right?” … [2]

Ø  Exploratory, or ad hoc testing, can be an especially useful testing strategy… [3]

Ø  "Exploratory testing is an interactive process of concurrent product exploration, test design and test execution.” “To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing.” James Bach, Satisfice (2001)

It is just to show up that different literature defines exploratory tests in a different concepts, people think different and defined their own perspective regarding topic. I am not trying to dictate the right concept, I am just telling you the most suitable concept to define exploratory test allowing for “exploratory” is pretty embracing word. 



... to be continued in next posts...

How software testing fits the cloud computing aspects?

Cloud computing is still an IT industry hot topic and has been gaining space and importance; Meanwhile, cloud testing remains unknown even for testing professionals around the world. It happens due to not only the limited documentation and references, but also because many companies already have a testing structure very well defined and does not have any application or product in the cloud. As mentioned before, much has been said about cloud computing, at this moment a question arises: How software testing fits the cloud computing aspects?

Let’s check below.

Cloud computing

Firstly we need to understand what cloud computing is.

In a few words cloud computing is a service to everything you want from IT industry. It means, store personal files, photos, documents and so on, where all files may be accessed anywhere and anytime with no software installation required.

Cloud computing is more than this simple example, which reflects your life as a user of cloud computing. The same idea is suitable to IT industry in totality. In a B2B (business-to-business) approach, it is possible use cloud service for an infrastructure, platform, software etc... In other words, everything-as-a-service (XaaS): infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS) and others.

A cloud computing advice would be: do not waste money in things that are not your business. In IaaS for example, it is not necessary to take servers inside the companies, they might rent as a service to fit infrastructure according to the business. Think about an online company that sells shoes; their business is to sell shoes, not to control technology for selling online, this is a typically scenario that consumes in a great way a cloud computing services.

The main idea of cloud computing is cost reduction and new opportunity.

Cloud testing

There is a subtle difference between testing in the cloud and testing cloud services. Testing applications that are in the cloud is a challenge on finding a test strategy to perform tests in this new computing platform; this approach will not been attended in this article. The focus here is present a cloud testing as a services that fits to software’s which are located in the cloud or not.

Cloud testing is a new way to manage your tests where companies offer testing as a service:

-          For Outsourcing: when another company implements your tests. The difference from traditional outsourcing remains on the fact that you get complete access on your repository to execute, get results or metrics and so on. Your pool of tests will be available to been managed anytime and anywhere.

-          For consuming infrastructure: when you manage your tests, to implement tests and to execute them. In this case, your repository is also available to been managed anytime and anywhere without demanding internal infrastructure. In additional you get access to several resources such as browsers (versions/platform), mobile (models/versions). This might been used for both automation and manual tests.

Benefits of cloud testing

The focus is cost reduction by consuming resources on demand. It means only pay for what really use. Some cloud testing providers offer cloud testing by hours per month where you get access to all platforms, devices and whatever the provider offer according to the contracted hours beyond offer follow-up of test creation and execution in a real time.

Another point is the security, in this case your company chooses for a dedicated storage in providers and off course, pay for it. This is seasonable for under development applications that are not in market yet or in case of some business strategy.

The great advantage on consuming cloud testing service, beyond price, is ease of access and virtualization environments without taking dedicated servers in house, making it simple, for developers and testers.

Conclusion

Your application does not need to be in the cloud to been tested in the cloud. This is a new way to test software in different approaches, such as, automation, manual, unit, black box, grey box.  To achieve a quality of software it does not matter where your software is going to be tested; the most important is apply best practices in software testing area. Cloud testing just makes life easier for tester’s professionals since it offers a several platforms and environment to test your application for a low cost.  The strategy to optimize cloud-testing services remains in the hands of the best professionals.

References

http://www.infoq.com/articles/testing-in-the-cloud-exploring-the-practice

www.informationweek.com/ blog/229206200

http://www.computer.org/csdl/mags/so/2014/01/mso2014010032.pdf

http://www.polteq.com/wp-content/uploads/2013/11/2013_EuroSTAR-Ebook.pdf