Is Your Self-Service Tough Enough? How to beat up on your system and find out.

[mwellens]

Is our self-service system going to be able to handle the demands of our employees actually using it?  How can I really tell?  I recently wrapped up a short-term assignment where I performed load testing on an SAP system landscape.  I always like these short specialty assignments as a change of pace.  In particular I appreciate doing a load testing assignment because it brings together such a diverse set of skills into one room to assess and solve a complex problem—find the bottleneck in system performance and make sure the users will be happy with its responsiveness.

This particular client’s system landscape consisted of an SAP Portal, ECC, and SRM system.  On the SAP Portal, Employees and Managers can execute HCM transactions like looking up total comp statements and viewing employee time and attendance while some power users could also create shopping carts and Purchase Orders on the SRM system.  Administrative users had direct access to the ECC and SRM systems to perform functions like hiring, changing time entries, posting GL, etc.  I was brought in a few months prior to go-live to use HP Load Runner to test the performance of the system.  I’ve outlined the breakdown of the activities conducted below and some lessons learned to share my recent experiences.

Design Your Load Testing Strategy

Prior to bringing anyone on board to conduct load testing, you will need to understand what it is you will test and how to best spend your time.  In your design you should answer the following questions:

1. What functions do you want to test?  You can recycle some of your scripts from different functional testing cycles to do this.  Make sure you cover all of the key critical areas of the system.  In our scenario we made sure to have scripts that logged users onto the portal to conduct both HCM and SRM transactions as well as a spread of different functions directly on the backend system.
2. Where do you think a bottleneck might occur?  In our scenario the client had a T1 line connecting on-site users to the different SAP systems as well as VPN connections coming in a different route.  We made sure to setup load generators on both routes to make sure each network could handle the traffic
3. Who are you going to need to execute this?  As I mentioned earlier, it takes unique skills to pull off a good load test.  The load generator specialist will be the person generating the scripts on the load generation tool.  This person will need to be familiar with in.  In my case I filled this role and had prior HP Load Runner experience.  Load Runner scripts generate code in C, so it was a plus that I also have experience in C. Since the scripts were run on SAP Portal and ECC it was a great that I am very familiar with these systems as well. We needed great ECC and Portal basis resources to make sure the systems were setup with proper load balancing ahead of time.  SAP security administrators were needed to make sure the test users we created for the load had all of the appropriate authorizations to run the script.  We needed functional consultants to generate test data and users prior to scripts being execute.  In addition, network specialists were needed to analyze the volume of requests coming across the network and look for bottlenecks.  operating system and hardware specialists were also needed to take a look at the performance of the machines upon which the . In our case a database administrator was never called into play, but this can be helpful if you suspect the database is where you are having issues.

Build Your Scripts

Once you’ve got a high-level design, it is time to get into the details.  For each of the functional scenarios you would like to test, you will need to know what steps you are going to take in the system and then record scripts in the load balancing tool that can be replayed to generate load on the system.  You can often re-use documents from other efforts in the project to help save time here and come up with the steps you need to take in your recordings.  For example, scripts created from your unit testing, integration testing, user acceptance testing, or training are great material to hand over to your load testing specialist to instruct this person on how to execute the different system functions you’ve chosen in your design.

Most of the fun begins when you start attempting to record the scripts.  Make sure you use the appropriate SAP server names and portal URLs that hit any load balancers you plan on having in front of your server to distribute load across your servers.  You will need to perform the recordings using such finalized urls/server names in order for this appropriate distribution to occur.

HP Load Runner makes recording SAP GUI scripts fairly easy.  You will first need to make sure that SAP script is activated on all ABAP systems.  It is then a matter of  hitting a record button and stepping through the screens.  You will have to make sure you record appropriate steps within the appropriate code blocks in Load Runner. Use the “Init” block for anything you want to only happen one time when the user logs on (such as the part of the script where you entered credentials). The “Action” block should contain the bulk of the transaction you are recording and will get repeated indefinitely for as long as you desire when designing your test execution in the HP Load Runner Controller.  The “End” block should contain the final steps taken when you want the user to finish up their session (such as closing all the GUI windows and confirming you want to end the session).

Any web-based script such as SAP Portal or web-dynpro ABAP are more time-consuming to record.  This is mainly because of how session management works in most web-based technologies.  In the case of my client, the programs opened up new windows and generated new unique session ids throughout the scripts.  Thus I had to code most of the steps within these scripts in the “Init” block since a new session would have to be created for each virtual user that would run.  Additionally, there is no “magic button” to press to find all of the session ids in a scripts and parameterize them all to capture and re-submit the appropriate unique hex-coded session id with subsequent requests.  There is a “correlation” feature in Load runner to help here, but it does not work 100% of the time.  I found myself having to write custom correlation rules to detect some of these unique session ids.   Don’t be surprised if you spend a lot of time here – it can take up to a couple days to record one complex script and get it to replay appropriately every time.  This is largely due to the iterative nature of trying to root out each session variable, re-execute the script, find the new failures, and the re-adjust your recording.

You can also parameterize your scripts with any other needed data that may change from iteration to iteration.  To do this you can create variables in the C code that is generated and provide tables of data to insert into these variables for every iteration of the script. This is helpful, for example, in performing a hire action where SAP often checks if the social insurance number of every employee is unique.  You cannot hard code entry of a single number because the system will require a unique one every time.

Execute your Test

Congratulations, you have recorded all your scripts!  You can execute them!  You will need to follow your design on which scripts to run, for how long, and with how many virtual users.  For HP Load Runner this is accomplished in the design tab of the Controller application. Once you’ve got all of designed you are ready to hit the “Play Scenario” button and watch your virtual users scurry to work pounding on your system landscape.

During the run, you will want to make sure no one else is using the system.  This will interfere with your results and probably also upset the real users who experience any reduced system performance because of the load you are generating on the system.  It is best to find a quiet time after hours to execute the test and first announce that the system will be unavailable and lock out any undesired users.

As mentioned in the design section, it is great to have all sorts of different specialists available during the test to monitor the different components of the system landscape such as application servers, hardware, and the network.

As the virtual users hit the system, HP Load Runner will output different real-time statistics about the load being generated and the system’s responsiveness to that load. For example, it will show the average response time for different code blocks and any errors encountered such as system timeout errors.

Fix the Bottleneck(s) and Re-test

Perhaps you were lucky and your system stood up to the load test with no issues.  Congratulations, you planned all of your hardware sizing and configurations perfectly!  However, this is not often the case.  Your specialists should help point you to any areas of the system landscape that faltered during the testing.  In the case of my client, we found many timeout errors during script execution the first time around.  We noticed the application servers and hardware were barely being taxed by the load test.  However, we did see that the network bottlenecked at one point where only a 1.5 MB connection was available.  We resolved to try boosting this area of the system landscape and then giving it another try.  This can often be an iterative process where when one bottleneck is fixed, it opens the flood gates to find another one down stream.  Once you’ve got all of the bottlenecks flushed out you should feel much more confident about your system capacity as you approach go-live.