Ins and outs of software testing for Communication Service Providers

July 2012

Getting testing right can make a big difference to both revenue and cost and many of the CSPs we spoke to during our interview process constantly referred to the need to reduce costs and the time to market for new communications products.

Testing service providers outlined how this can be achieved, for instance, through centralization and off-shoring of some testing functions, placing a greater emphasis on requirements planning before test planning and execution begins, and critically, by reducing the rates of software defects in the production environment (that is the real operating environment rather than the controlled “test environment”). Based on our discussions with testing service providers, it is especially important to reduce errors in those systems that have the most direct impact on a CSP’s financial and operational performance.

Darren Coupland of Capgemini-Sogeti’s Global Testing Team, and currently leading a major testing and ALD transformation program for a tier one telco player comments: “It’s no longer good enough to just detect defects during the test phase and prevent production incidents; that is a given. Increasingly CSPs now demand that test providers really focus on stopping business-critical defects being introduced in the first place by taking a Shift Left approach. This isn’t anything new, but it’s always been difficult for test teams to generate the necessary traction and influence ‘upstream’ changes. But with a current client, we have developed test transformation programs, with quality improvement at their core, that combine advanced governance, metrics and accelerators, and our own thought leadership & IP, with aim of not just transforming the testing service but also the quality of wider project deliverables that have a direct impact on the client’s business goals.”

When CSPs were asked to prioritise the importance of systems to be tested, billing emerged as the most important system to test thoroughly (see table 1). Around half of CSP respondents singled out the billing system as most critical from a test perspective.

Critical software applications, such as the billing system, can be a source of major headaches for CSPs: functional limitations, or the reconfiguration effort required to support changes to products or the introduction of new products, mean that time-to-market can be long: testing is an intrinsic part of reconfiguration or functionality improvement projects.

Table 1: Systems described as “most important to test” by CSP interviewees

Software Testing

Nevertheless, some CSPs told us they believed that no one system was more significant in testing terms than any other. Rather, it was more important to conduct system integration and end-to-end process testing, as the greatest number of errors tends to occur in the gaps between individual systems.

From the discussions we had with CSPs in relations to how they measure the success of a testing program, we can conclude that CSPs take different approaches to measuring success. Some have a set of very detailed metrics that cover all aspects of a test process, while others simply say that reducing the number of software errors going into the live production environment is the only significant measure of success. Table 2 indicates the ranking of those measures mentioned to us by interviewees.

Table 2: Metrics used to judge the success of testing programmes

Software Testing

Many of the test managers and software quality assurance specialists that were interviewed said that software testing was increasing in importance and becoming more challenging, and that the ways in which it was carried out were changing, in response to a number of specific drivers:

The four key drivers which were identified through our discussions with CSPs, increasingly impacting their telecom software testing needs are:

    • Increased competition and the need for innovation and speed to market
    • Rising complexity of services and IT
    • Growing mobility and the rise of user applications
    • Sustained cost reduction imperatives

Software testing for CSPs market revenue reaches US$1.7 billion

The study also defines the current and forecasted total value of independent software testing for the CSP segment. Independent software testing for CSPs has been enjoying, and will continue to benefit from growth for a number of reasons, with the primary driver of market expansion being the continued strong investment in telecoms software itself. The global telecoms software market as a whole is expected to demonstrate a CAGR of 6% between 2011 and 2016.

Meanwhile, the services and applications that service providers are delivering to customers are becoming increasingly complex. They are also being delivered across a much wider range of fixed and mobile devices including different varieties of feature phones, smartphones, tablets, laptops, PCs, IP phones and TVs. This means they are harder to deploy and configure in the first place, and the task of ensuring that the customer experience remains high is becoming more intricate and multi-faceted.

This creates a need and an opportunity for an independent testing organisation to ensure that everything works end-to-end, from the provisioning of a service, through the back-office systems needed to support and bill for the services, to the end user application itself. Much of the market growth is coming from the emergence of this end-to-end testing of applications - with testing processes that span everything from the OSS/BSS systems to end-user devices. These processes are bringing network and device testing into a field that has traditionally been considered separately from the OSS/BSS stack.

As defined in our market sizing exercise, telecom software testing revenue is estimated to have reached US$1.7 billion in 2011, while software testing revenue is estimated to have reached US$12.7 billion. This means that the CSP segment generates around 13% of the software testing revenue on a global basis.

Software Testing

When it comes to the top independent software testing providers, all except one are global consultancies/IT solution providers and business process outsourcing specialists. At the end of 2011, the top 5 suppliers of software testing to the CSP segment, defined in terms of their total market share, were:

    1. Amdocs
    2. IBM Global Business Services
    3. HP
    4. Accenture
    5. Tech Mahindra

Amdocs is the lone representative of the OSS/BSS vendor community in the top 5 independent telecom software testing group, as it provides testing services to CSPs for software beyond its own portfolio.

To achieve success, CSPs must provide a consistent, high quality user experience across all customer touch points, efficiently and with a low cost of quality. This is driven largely by the way testing is managed, planned and executed,” said Gil Briman, Vice President, Amdocs Consulting Division and head of Amdocs Technology Integration Services. “Amdocs has developed industry leading testing methodologies and expertise that can be applied to our customers’ entire IT environments, including both Amdocs software and third-party applications. With a team of testing experts across 30 geographical locations, Amdocs Testing Services has become the leading independent software testing provider in the communications market.

Further analysis of the software testing revenue for the CSP segment can be found in our report: ’Software testing for CSPs – Market Analysis’, together with drivers and trends impacting customers’ expectations and the competitive landscape of the suppliers in the CSP market.

More information on this report»

An Executive summary of this report can also be downloaded»

For more information on the Telecom Software Testing for CSPs market or any other project in this segment, please contact Isabelle Paradis at:
+1 514 270 1636