At the recent SAP TechEd conference in Las Vegas, michal.krawczyk2 from Int4, a fellow mentor, conducted a session (NET38702 - which will be repeated in Teched Barcelona), running test scripts in SAP PI/PRO, and I was amazed by how many people showed up! At first I couldn't understand why such a huge number of people would attend -- it's not like testing is a very exciting topic. But when the discussion got started, I started to get it.
Despite the fact that testing is a problem that confronts everyone in the field, very few people are working on testing tools. In other fields, there is a culture of test-driven developments, but this isn't something we have done much in PI/PRO. So TechEd attendees came to learn about what is being developed in the area of SAP PI/PROtesting.
The basic problem is that although we want to be able to do testing, current methods are inadequate to cover the use-cases that exist. When Michal Krawczyk asked for a show of hands of how many people were working on testing tools, only a handful of people raised their hands, so I thought I'd share their research in this post.
One group of developers was conducting manual tests. They had saved a couple of test cases as test instances in Message Mapping or maybe Operation Mapping, and once they had made changes, they would call up these messages to see if they would still run. The problem with this approach is that since you are using older messages that haven’t changed much, it is difficult to catch the unknown changes you don’t know to look for. This makes for a somewhat limited test, but it will still give valid feedback if the work is correct.
SAP PI specific testing tools
My company, Figaf, has also made a SAP PI testing tool that makes it easy to record and test messages from the sender module to the receiver model chain. I believe the challenge with these kinds of systems is to find a way to catch the message and test it all the way through the process, while keeping the method as simple as possible. Figaf meets this challenge by adding two modules to our production system, which allows us to replicate the traffic on that development system, so we can check that all modules and routing capabilities stayed the same.
With this tool, we're able to get a lot better test coverage. In ten clicks, I can set up a test case for a hundred new documents and be able to process them in the easiest way. In contrast, users of some of these other tools might need to use a third party system or work through various different steps before being able to run the tests. At this point, the assertion the system receives is the original output document, so I can expect the same on my test system. Then, as you proceed, you can ignore some elements (for example, if we know this timestamp is different, or this system name is different), so you only see the important differences. The tool supports comparisons in EDIFACT, X12, Text and XML. It does not test adapters or backend systems, but it does make it a lot easier to test all mappings, modules and routing in your system.
IRT can even be used for free.
Michal Krawczyk and his company Int4 have developed an interface regression testing tool - IFTT which is able to regression test both SAP PI/PRO as well as SAP backend system interface (user exits, ABAP enhancements). Due to the fact that the tool is tailored for SAP landscape you can easily create test cases from any existing IDOC or ABAP Proxy message by selecting the message ID (IDOC number or GUID) only. There’s also no need to create any assertions as IFTT is comparing the data on the SAP database level (using SAP tables). By utilizing this tool both the developers and anyone responsible for interface testing should be able to catch any unwanted change before it makes it to production landscape, allowing them to pinpoint a potential production problem. Michal Krawczyk said they at one customer they have tested 30 different scenarios with total of around 10.000 different test cases (messages). Of course, that's a bit of a different scale than if you're just sending in five or six different test cases per scenario -- with this you're going to cover a lot more ground and be sure that the mapping and the whole interface on the SAP backend system is working.
Other tool was:
Some other people were using XMLSpy from Altova to do some tests on their server. I think you need Enterprise version to do tests this way.
Another guy I spoke with was using Ready API from Smart Bear, the guys that are making SOAPUI that we all are using. With this tool, they are able to set up test cases; for example, it allows a bit of XML to be sent through the system, then downloads the message, and generates an assertion that allows us to specify the values we want to see.
One group is using a Computer Associates or HP testing tool to conduct end-to-end testing. This is how it works: First, log into the third-party system, pressing all the keys to generate a script that goes through the screen; next, fill in the relevant data, and finally, complete the transaction, and the message should be sent through the PI/PRO to the ERP system. At that point, make an assertion that the order was created, and the system should return a total amount, etc. This method would cover the full process end to end.
One of the challenges with this kind of tool is that it requires a lot of time to set up the test cases. This means that whenever you want to do a test, you first have to go through all these test cases, make sure they match, and update them. Only once they're running well, are you ready to start the actual test. Another potential problem could be that since you have to go through so many steps, you may limit the complexity of the example transaction (for example, you might not create that complex purchase order), or the number of tests you run (you might run only one or two cases, when you really should run ten).
There was also a coding session by SAP showing how they were doing their regression testing. In this case, they had a virtual environment, which means they were able to create scripts that were built in the PI system from scratch. These scripts were deployed to an Amazon Cloud instance, which could then be used to run different scripts, like adding or calling the webservice and getting a response back. Developers could then gather data on how well that task had been performed, how long it took to do ten calls, then a hundred calls, and so on. They were also able to vary the size of the message to see how well the message scaled on all different levels. This is obviously helpful for testing system performance, because it allows testers to ask questions like 'Did we add anything that just makes the database expand?’ or ‘Did the performance to suffer?’ For most customers, this method would not be particularly useful because they would first need to set up the framework and collect the business cases.
I have heard of a system from X-Case that is doing something similar, though I haven't tested it to see what it can do or how it works. Also, IBM purchased Green Hat now Rationale Integration Tester, which is one of these more complicated environments to set up and run, I'm not sure on how well it works with PI/PRO.
The the last tool I want to cover is one created by my old colleague mortenwittrock about 10 years ago, called the Detroubulator. This tool was able to run message mappings, but it was really cumbersome to set up all the test cases.
So that wraps up the list of SAP PI/PRO testing tools that I wanted to cover in this post. If you have other tools you are using, please share them so everybody else can get a better idea of how you are testing and what you are testing. If you are looking to perform tests on your own system, do check out some of these tools -- and let us know which ones helped you the most.
- SAP Managed Tags:
9 Comments
