In a world of rapidly changing business challenges, one thing remains a constant driver for success: Quality. At SAP, we strive to deliver the highest quality software solutions for customers ranging from small businesses to large enterprises across many industries and around the world. This commitment to quality extends throughout the SAP portfolio of products. That’s why we reached out to our Business Intelligence customers and gathered feedback on previous releases of our Business Intelligence software. Based on the feedback, we continually investigate methods which improve on our quality processes. Quality improvements have resulted from large-scale development methodology shifts all the way to individual product teams continuously improving their day-to-day processes. In this article, we will go over some of the improvements implemented on the SAP BusinessObjects Enterprise product line.
Switch to Lean/Agile Development
Starting from the BI 4.0 FP3 project, there needed to be a customer centric shift to our organization. As such, Lean/Agile development was adopted as the standard software development process. The diagram below provides a bird’s eye view of the quality cycles within a release. Note that in the scope of this article, we only discuss the high-level approach.
The release cycle is divided into monthly sprints, where three sprints are grouped into one wave. Incremental functionality, for a particular feature, is developed and tested within a sprint. At the wave boundary, the collection of incremental functionalities developed within the sprints constitutes to a feature which has been fully developed and fully tested. Depending on the scope of the release, multiple features are in development at any point in time.
Daily product acceptance tests are performed, using automation, to ensure that core workflows remain enabled for development to continue. At the end of each sprint, a suite-wide acceptance is executed to ensure that the projects quality targets are being met. Following the lean/agile concepts of “pulling the line” to maintain quality, new feature development cannot resume until the quality criteria has been achieved. This allows us to reduce the late stage effects of having a large scale effort to stabilize the product at the end of the release cycle.
As the release progresses, more emphasis is placed on stabilizing the product versus new feature development. A hardening wave is employed at the end of the release cycle to complete further tests such as Customer Focused Deployments, Languages, and Platform support. This wave is also used to resolve the remaining outstanding defects.
Quality Criteria
The following table provides a summarized view of the quality criteria achieved at each sprint, wave, and hardening wave. A detailed example can be found in the appendix.
| Target | At the end of each... | Criteria (summarized) |
|---|---|---|
| Usable | Sprint |
|
| Potentially Shippable | Wave |
|
| Shippable | Hardening Wave |
|
Further Quality Enhancements
In addition to the shift to Lean/Agile development, the following are some of the quality enhancements that have already been implemented or are in the process of doing so.
Defect Thresholds. The concept of defect thresholds was introduced to prevent a large accumulation of outstanding defects. If a product team crosses the threshold during the development cycle, all new feature development is halted until the defect backlog is back to a manageable level for that team. Thus, quality is maintained throughout the development cycle and it minimizes the chances of defects being pushed out of the release.
BW Quality Task Force. Formation of a task force focusing on improving our BW Quality. The objectives focus on three main areas: Target consistent behavior across all the suite of BI clients; improve design-time and run-time performance of all BI clients with BW as a data source; lastly, improve general functional quality of BW integration.
Customer Scenario Testing. By evaluating key workflows and environments employed by our customers, we identify and close gaps in our test strategy. Also, a cross representation of complex customer environments (including data) are replicated in house. According to the test strategy of a release, a set of acceptance tests are executed on each environment to ensure no regressions have been introduced.
Integrate Customer Escalations into core testing workflows. Similarly to customer scenario testing, we review escalations raised by customers. Any gaps identified in our test strategy are supplemented by including further tests to minimize the possibility of re-introducing those escalations in further releases.
Partner Testing. We extend opportunities to our partners to access and test our upcoming releases. Partners are able to evaluate and learn more about various features via guided exercises or free form activities using their own supplied data. Partners provide us with feedback on the solutions which is then evaluated by the development team for inclusion in the release.
Going forward
As described, numerous changes have been made to our processes to better product quality. In the spirit of continuous improvement, we are further analyzing our processes via retrospectives held at various levels of the organization. New improvements are then prioritized and incorporated into upcoming releases.
Appendix
Example of Usable Quality Criteria:
| Criteria | Metric |
|---|---|
| Functional |
|
| Performance |
|
| Reliability |
|
| Product Standards |
|
| User Stories |
|
| Scrum |
|
* NOW Defect: Issues in Major workflows that would impact greater than 90% of customers. These defects must be fixed as quickly as possible.
- SAP Managed Tags:
2 Comments
| User | Count |
|---|---|
| 14 | |
| 11 | |
| 10 | |
| 10 | |
| 10 | |
| 9 | |
| 8 | |
| 7 | |
| 7 | |
| 6 |
