Almost every customer visit for someone from the IMS teams starts the same way: a customer incident with very high priority is created and a processor is assigned to work on it.
This time the story was not very different. One day I received an incident from one of our large customers. The issue was quite tricky as it was concerning the cluster communication of the quite large AS Java cluster (5 instances with 3 server processes each) installation. The problem was actually so severe that it caused a complete cluster restart which endangered the core business of the customer. As one wise man said, if the SAP software in the world disappeared, all companies would stop working.
As you see there is no pressure :wink: . Even in the darkest of places there is always glimmer of light. Despite the pressure, the phone calls, the gigabytes of traces and all other information, you always have colleagues that are backing you up. There was an amazing team of SAP employees on-site that was helping me reproduce the problem, collect traces and keep the customer calm. In the end we found the root cause of the problem quite fast and we patched it even faster thanks to my great colleagues from the QA team in IMS Bulgaria and the final assembly team in Russia.
The customer was happy and everything was looking as if we could close this incident like every other. But this time I had a feeling that this wouldn’t be the end of the story. 2 months later my manager and I received an e-mail saying the customer was requesting an on-site health check service and I would be the best fit as I had already worked on the incident.
All the arrangements were made in the blink of an eye and before I knew it, the team and I were all flying to South Korea to meet with the customer. I flew approximate 17 hours longer than the rest as they were all working in SAP China. Although I was very tired from the flight as I’m quite large (in all directions :wink: ) and the seats are quite small, I was not at all sorry for the two hour bus trip to my hotel that followed. Seoul mesmerized me. It was a mega polis that was so well organized that my zero knowledge of the Korean language didn’t get me lost. Everywhere I looked, there were skyscrapers, highways with 4 or more lanes, and everything was extremely clean: you could see the sky, there was no litter on the streets, and there were no funny smells, something not typical for large cities these days.
At 8 am next day we arrived at the customer. The first thing that astonished me were the buildings that the customer was occupying: 30+ story skyscrapers made of glass and metal. Since I started working for SAP I have visited several customers, and some of them had even bigger headquarters, but this was the first customer that had its developers working in a shiny skyscraper. Usually the developers and the administrators are working in something like a market hall filled with cubicles.
It was not easy to go into the buildings. There were several security checks and they gave us special access cards that could only enter one specific floor, but not before we confirmed our identity by showing our passports. We also had to write down the serial numbers of all personal electronic devices.
After the security check we entered the building and the room that was going to be our work home for the next five days. First we met with the teams that are responsible for all SAP AS Java systems at the customer and their respective managers. Although they all understood English and spoke it quite well, we had a personal interpreter from English to Korean. We all felt like we are in a G7 meeting. At first it was very strange, but you get used to it very fast and you start to appreciate it.
The team that we were working with was great. During the first meeting they depicted a complete picture of all their systems and their business usage. This is quite often overlooked and forgotten, but without that information you could not correctly analyze the systems and determine if their sizing is correct. During the next meetings we had special sessions on different topics: analyzing Out Of Memory (OOM) errors and thread dumps, how the central services of SAP AS java work, Cluster communication, core AS Java services and their tasks in the application server, etc. During all session they asked direct and very concrete questions regarding the content and they did not stop until everything was clear. And from a trainer’s point of view I can say that this is one amazing quality of the Korean people. Usually when you are training a group of people, you ask them questions regarding the content in order to find places where something is not clear. This has a negative side that you cannot check everything, but only some parts that you think cannot be easily understood. When people ask, you get an immediate feedback where the content is not clear and adjust your explanation.
The first four days continued in a repeated fashion. During the day we had knowledge transfer sessions, searched for issues on the customers systems, and analyzed already found issues. During the evening we went all together to a restaurant or a bar and had unfamiliar delicious food and some beer mixed with Soju (distilled beverage made of rice, wheat, or barley). During these dinners you can get to know the people that you have been working with the whole day and this makes it a lot easier to work with them during the next days, weeks, years.
After the first four days, when all knowledge transfer sessions had finished and all issues were identified, I started thinking about how it was to get out of my comfort zone and about all the new things that had happened during the last 4 days. But even better, I got to see how our customers use their systems, what pain points they have, and what frequent and repetitive tasks they have to do that takes up too much of their time. And this information is priceless as you can give it to the product owners and make our products better suiting the customer’s needs.
Visiting a customer can always look like a scary task, and to some extend it might be, but you get so much in return for going outside your comfort zone: see how our customers use our products, see the pain points of a product, meet new people, see new places, eat thinks that you don’t know what they are, … At the end you get in return as much as you have given to the customer.
- SAP Managed Tags:
