With more than four thousand employees, as well as its own postal code, News International is one the largest media companies in the world. A division of News Corporation, the UK-based company is most known for being publisher of The Times, The Sun, News of the World and The Sunday Times.

In 2010, News International took the bold and pioneering move to move to a ‘paid for content’ business model for a number of its digital titles. This brave move would include the introduction of an iPad version of The Times and The Sunday Times. On that news, Christian Henderson, the company’s Head of Test and Release Management, began searching for an efficient way to test the functionality of the iPad apps under real-world scenarios.

Researching his options, Christian concluded that traditional QA agencies and outsourcing firms would not be able to match in-the-wild testing is terms of cost, time or ease-of-use.

“The use of contractors from our PSL works well in some areas of the testing lifecycle and products,” he said. “But in this instance we needed specific skill-sets. We required a more defined, focused, streamlined and experienced, low-cost test solution acting out real world’ scenarios."

Testing Set Up and Execution

Needing to cover as many angles as possible, Christian was pleased to learn that  testing in the wild makes it easy to test their app by device, OS version, location, carrier and other categories. With that in mind, Christian would upload specific test cases designed to reflect the actions of their real-world user base. Testers were instructed to focus on:

• Crash defects
• Subscriptions
• Downloading sections within an Edition
• Continuity of user journey
• Wi-Fi, 3G and edge connections

Test Cycle Results

Christian and his team were amazed at the pace and accuracy of the bug reporting with this method. Within hours after opening his test cycle, Henderson was already seeing multiple issues being reported. But it wasn’t just the quantity of bugs that impressed him.

“The quality of bugs found was terrific,” said Christian. “By our second test cycle, testers were submitting videos and screenshots of issues we had been verifying in-house in addition to those bugs we had not yet found ourselves. The response time, effort and attention to detail was excellent. We’ll continue testing our products under real-world scenarios.”

As News International develops more applications for the iPad and other devices, Christian expects real-world testing to play a major role in future projects.