chris.kernaghan

SAP Music Playlists - an experiment

Chris Kernaghan — Tue, 21 Aug 2012 14:45:46 GMT

At the start of the year I attended Monkigras which increased my fascination with data sets and relationships between them. Over the course of the year I have been looking at different data sets, how to measure and quantify things as well as how to related different data streams to find correlations or draw conclusions. Part of this journey has led me to #DevOps and the pursuit of improving the project delivery, the other part of this has led me to discover what data can I measure about myself and what correlations and conclusions can I draw about my body's performance versus what I consciously decide to put it through (A whole other blog post in itself and perhaps not for SCN)

So sitting here listening to some of my favourite music, I decided to run a small experiment based on this presentation from Matt Biddulph's Monkigras' presentation- I think for now it would be interesting to see what music people listen to and see if we can find any relationships against job functions and age. If we get lots of entries, then it would be possible to generate a LastFM playlist by SAP area or just a general SAP playlist :-)

In order to take part, I would like people to comment and enter their top 5 songs they love to work to (not their favourite songs) their job function and whether they are fall into the following age ranges - 20-29, 30-39, 40-49, 50-59, >60

As you all know, hopefully, I am a NetWeaver Technical consultant (Basis) and I am 30-39,

my 5 songs are

Linkin Park - Crawling

U2 - Beautiful Day

Florence and the Machine - No light, no light

Ellie Goulding - Lights (Bassnectar remix)

Foo Fighters - Best of You

Looking forward to seeing some great data

Chris

Landscape Virtualisation Manager concerns

Chris Kernaghan — Thu, 27 Oct 2011 04:17:58 GMT

Last week I received some disturbing news about the license model of the new Landscape and Virtualisation Manager (LVM) which is entering Ramp-Up in November, before I get to the news I received lets look at what the LVM is.

The LVM is a new product from SAP which is the replacement for the Adaptive Computing Controller (ACC), the new LVM has increased capabilities over and above the ACC – for example the LVM has the ability to script and execute system copies automatically, it has dashboards and lots of management capability of physical, virtual and (hopefully soon) Cloud environments.

The LVM is, for me, one of the most exciting SAP products coming out in the next year, it effectively ‘closes the circle’ of the SAP technical administration tools – Landscape Management Database, SMSY (System Landscape) and System Landscape Directory.

By ‘closing the circle’ in terms of Technical administration I mean, the ability to have multiple sources of information cross-feeding each other efficiently providing a single version of the truth for each of the administration applications

LMDB – Analogous to the SLD, it provides many of the same functions and synchronises directly with SMSY

SLD – Provides information on each registered system, providing software component and patch levels.

SMSY – This is the central hub of all information in Solution Manager, everything that is associated with a system gets it’s information from here

LVM – Provides dashboards, control capabilities for instances like start/stop or relocate

During Teched Bangalore, a colleague of mine was attending a Virtualisation session, during this session it was mentioned that the LVM will be a licensed product and will not be provided as part of the SAP license like Solution Manager. This to me was a vey strange statement as it was always my understanding that the LVM, like the ACC would be provided as part of the SAP License and would be available to all for download. For me, not providing it in this fashion would be a bad idea for the following reasons

1. No-one will use it.

The ACC has taken many years to get to where it is today, and it is a far more useable product in the last 2 versions that it ever was before. Still there has not been great uptake for it with customers, again for a number of reasons (integration with SolMan being one of them), but at least the ACC was free, this encouraged people to use it, even if was a skunk works project by the Basis team after seeing it demonstrated. If you make people pay for it, and get the price wrong then you alienate your market. Also how do you really quantify the ROI in saving the Basis team nearly an hour when doing a kernel upgrade across 15 servers. SAP have been promising for years to make administration easier to reduce TCO etc…, now that they have delivered tools like Solution Manager 7.1, the LMDB and the LVM, those statements have never looked so attractive or achievable.

2. It will not function within the partner ecosystem

One of the key selling points for LVM is both the extensibility of the product to link up with different infrastructures (see top diagram), it will not replace your Tivoli or HP equivalent, but work with them in a push/pull fashion. Partners will provide good resources if there is a demand from customers, they will just provide plain resources if it’s contractual. If no-one uses the product, then SAP can expect to see poor partner development of add-ons for the product which would make it a killer application.

3. Value proposition disappears

One of the many things that SAP have touted within the LVM is the ability to run system copies and refreshes, for this capability there was an expectation of paying for a license – which was reasonable. The main value proposition is that by using LVM, and with it’s tight integration into all the landscape management components mentioned above, the whole management of the pre, during and post tasks was infinitely simpler. If the whole LVM incurs a license fee, and the partner ecosystem falters, then the 3rd party tools, which handle more than just SAP start to look attractive again and SAP will have developed a smart application which no-one uses.

Today I have a call with SAP to get to the bottom of this and hopefully I will be happy, although probably under NDA so will not be able to write about it until ramp up. Regardless of what SAP are going to do with the product, I would strongly urge you to look at the product – it is a great piece of technology and does ‘close the circle’ on technical administration. If SAP treat it right and nurture both it’s partners and the ecosystem, then LVM can grow into a cornerstone product for SAP applications, if SAP treats it’s customers as a way to make a fast buck out of licenses then SAP will have wasted both money and brownie points with the #sapadmin community. SAP will have pods for the LVM at Madrid and also check out session TEC120 for more information.

What can you do to keep up with TechEd Las Vegas when you cannot be there

Chris Kernaghan — Mon, 12 Sep 2011 00:37:46 GMT

Last week Thorsten Franz (@thorstenster) wrote about why he attends SAP TechEd at every opportunity, after reading it I thought 2 things,

1. I am jealous as hell that my SAP Mentor brothers and sisters are out in Vegas and I am not there,

2. No-one has talked about how those of us left behind can take part in SAP Teched - so here we are.

In recent years, SAP has embraced the social aspects of conferences using YouTube, Twitter, Facebook and LinkedIn, which I love, it shows how much SAP has changed and wants to engage with it’s customers – getting them to listen is still a challenge but they’ll have the conversation. I remember last year when I went to Las Vegas and was contacted by Gunther Schmatzalf through the social connections provided by the TechEd website, it was a productive meeting and completely facilitated by social media.

In addition to all the social connection stuff, SAP also stream a great many sessions online, so for those of us left at home we can watch the sessions on-line (Virtual SAPTeched) – it remains to be seen as to whether people will be able to ask questions in the same way they were for the Virtualisation week in April, but I will be hoping.

For me I will be taking part in TechEd as much as possible, being in the UK the time zones work in my favour as many of the sessions will be at the backend of my business day so I can usually watch them in peace. Also I will be keeping up with my friends at TechEd using twitter and e-mail, during the week ASUG produced a great list of the best people to follow on Twitter for TechEd Insights (ASUG News.) As usual there will be a lot of hashtags to follow on Twitter, this link will show you the major hashtags, and there will be some unofficial hashtags used between friends which can yield some very good information too

Like all things, you will get as much out of it as you put into it, the information and session information is out there it is up to you to make the effort to string it all together and get the information from it. So good luck and happy engaging :-).

Trial Cutover 2 (TC2) completed and the lessons we learnt from it (CUUC Part 6)

Chris Kernaghan — Thu, 08 Sep 2011 08:55:34 GMT

Christmas was long over, the January blues were closing the month and I had just spent another weekend at the Hilton at the Birmingham NEC performing TC2 - which went very well. So I have decided to spare you the details of the process - instead I wanted to document how the analysis of TC1 influenced the upgrade process and if those tweaks worked.

So what did we 'tweak' between TC1 and TC2? The table below details the items discussed in the last post and the updates from TC2

As you can see from the table above we had several changes to how we executed processes as opposed to changing the processes themselves - this was to provide the consistency that is necessary when coming to the later stages of upgrade projects.

In terms of timings the tables below show how TC2 ran compared to TC1 and the PoC CUUC process, and how our mitigation techniques worked for us, especially on the Unicode conversion

So as you can see the overall time has come down quite dramatically in both the Upgrade Uptime and the Unicode phases, this is primarily due to the index defragmentation done on the database over Christmas and our own improvements in running tasks.

As well as technical measures to improve performance we also implemented a number of soft measures around people and upgrade management.

The shift patterns were improved, we found that the other upgrades were running at paces which were not accurately reflected in the plans - this caused problems for the team as I wanted each person to have a 'buddy' to check upgrade inputs and reduce errors.
The previous post talked about the Cut over communications, these were difficult over TC1 because the shifts were not optimal. After reworking the shifts we found that the communication methods worked quite well, but communicating with a geographically distributed team across timezones is not easy with so much at stake.
SAP Streamwork was a very useful repository for the upgrade documentation, the team was able to keep their documentation up to date and available to the whole project team. I also put a daily upload of the project plan on Streamworks so my team could keep up to date with the plan revisions as they did not have file system access to the client.
The technical team did a project plan walk through, counting out each hour and detailing the tasks they would perform, this is difficult one to call as to it's usefulness. My team found it very useful but it does take about 4-5 hours to get through, on simpler projects I am not convinced of it's value.

Coming out of TC1 and TC2 was a recurring issue around both backups and the fall back scenario, both of which are vital for a customer to ensure they can Return to Operation (RTO) as quickly as possible. The main issue that we had going into TC2 was the actual scheduling of the post-Unicode backup, this is a vital checkpoint as the system has to be re-introduced into the backup schedule as soon as possible and also needs to capture all the changes of the Upgrade, Unicode and transports. In order to do this we decided that we would run an On-line backup of the system as soon as the transports were imported and before an SGEN was executed. As a result the backup would be running whilst the users were testing the system and because an SGEN had not been run they would experience degraded performance, this was going to be something they would have to live with but creating a recovery point for the system was more important than a temporary performance issue.

The fallback scenario was another challenge, during the project the client had a requirement for an additional Pre-Prod environment to allow testing of a seperate project stream which was falling behind and would not go-live with the other work streams. At the time, providing this system was a major pain and caused a great deal of stress, but it would become a great opportunity for the fall back. The diagrams below shows how we used the additional Pre-Prod system to restore the Pre-Upgrade backup on to it and in the event of a fall back re-present the storage to return to the previous version.

In the event of invoking a fallback, the process would change the SAN presentation of disks to re-present the restored database back to the Production source and allow a much shorter RTO than a straight restore from tape.

Now that we had the database backups and fall scenario constructed, all that was left to do at this point was to get ready for Production in a few weeks time.

Data analysis of CUUC outcomes of TC1 (CUUC Series part 5)

Chris Kernaghan — Mon, 18 Jul 2011 01:59:17 GMT

As regular readers will know, my team and I have completed The 1st Trial Cutover (CUUC Series part 4) and we collected a vast amount of data. We also blew our time budget of 60 hours out of 96 hours total downtime, but we recorded the areas where the overruns occurred, so now I will take you through the analysis of what we found and what mitigations we put in place.

As a veteran of these type of projects, I have done this analysis many times, and I find that it is better to structure my thinking into Constants and Variables - for example

Variables: things we can change - for example, the number of work processes

Constants: things we cannot change - for exmaple, the time a backup takes

During the Upgrade and Unicode processes data is flowing through the system at phenominal rates, it is important to benchmark these data operations/flows to provide performance analysis - thank fully most of this work is done for you and it is simply a case of working out how to use this data to give you the best analysis and edge for your CUUC process.

We have seen that the Upgrade provides a file called UPGANA, which gives a high level breakdown of the Upgrade phases and their timings. The next important data set to be analysed is the Unicode conversion - the Unicode conversion, like the Upgrade, writes a great many log files which can become quickly bewildering, the important thing is to start at the top and work your way down. In order to do this, you start with the Export_time.txt and the Import_time.txt, these files provide you with your package performance details, like Operation duration, Size, Data Throughput.

The table above shows how specific packages performed during the Unicode export and the time period they were executed in. This can be very important, as the Unicode process should be viewed as a holistic process - everything running on the system/servers has the ability to affect other things. So keep an eye on things like R3load processes accessing the same cluster table at the same time and causing performance issues. The text file is also represented as an HTML file with nice graphics, which if you are inclined you can merge together to give a graphical representation of what packages run when and for how long - I was not so inclinded and did not. These files and their contents are also provided for the Import processes.

So now that we know where to find the information and the formats that information takes, let's dive in and see what variables exist which can be used to affect change in the Upgrade and Unicode performance as shown in the previous post (The 1st Trial Cutover (CUUC Series part 4))

From the table and the details above you can see the variables we can affect and the places where we can find some of the data which will influence how we change the variables. There is a wealth of deep dive analysis we could do, like running performance traces which will highlight the CPU, Memory, Disk I/O at every second throughout these processes - Jim Spath's excellent Unicode blog series shows the type of deep dives you can do into data sets. Personally I am not a massive fan of doing this, unless I really have to, I find that the analysis can become a goal in itself and distract from the overall goal of getting the thing done!

The 1st Trial Cutover (CUUC Series part 4)

Chris Kernaghan — Mon, 11 Jul 2011 00:25:07 GMT

So at this point we were 6 months into the project, my team had completed the PoC, Dev and QAS upgrades on all the applications - working with the new hosting provider as they conducted the data centre transition.

During this time we faced many issues around performing Upgrades and Unicode conversions using Parallel Export/Import on VMWare systems (that is a subject for another day), as a result of these issues I had a nervous client who was not convinced that we could pull this off within the downtime window. This upgrade (designated TC1) was to be the proof of the pudding, the last 6 months of work and preparation was going to be in full visibility and it had to succeed within the window or have very good reasons why it did not.

As my team got closer to the upgrade time, we refined the plan more and more - I had worked with my project manager on a previous project and knew he was an absolute whizz at MS Project, but he outdid himself this time. The project plan we had was based on ASAP, with nested plans and cascading dependencies throughout. This made it very easy to update the top level and see the knock-on effects of slippage. Finally I conducted walk throughs with people to make sure the timings were reasonable and that no-one was staying up too late.

As we had 4 days downtime, I had a technical time budget of 60 hours to do all the upgrade work across all the applications. So we worked out a shift system which enabled the 6 people on the upgrade to have rest periods during the upgrade.

It is important that the responsibility be shared among the team members, but utimately there has to be a single person responsible for each application. That person has to be the expert in that particular system during upgrades, if there is an issue they have to be involved in the troubleshooting, although they do not get the final decision (that's the team lead's repsonsibility) they are pretty close to it.

The table below outlines the known issues that we had going into the trial cutover and the measures we had developed to try and overcome them.

Apart from the known technical issues, we also had non-technical issues

In every upgrade it is vitally important to be able to guage the performance of the upgrade so that you can predict things like, are we on course, do I need to reschedule people, what is the report to the project, when can I sleep. To help with this, SAP have provided an excellent analysis file - called the UPGANA.xml file, you will find it located in the HTDOC\ directory under DIR_PUT. As you can see from the picture below, it has much of the top level information -

It also has all the timings for the upgrade phases, which is vital for keeping an eye on your general upgrade performance, you should use the log files for specific timings during the process, but I find this file works well for project managers :-)

The table below shows 5 of the longest running phases in the Uptime and Downtime part of the PoC and TC1 upgrades for comparison.

As you can see, the PoC was a much slower upgrade, and for the risk adverse among you, you are probably wondering how I managed to keep my client on my side with timings like those of the PoC (I'm not telling.) The TC1 times show that the PoC was massively under powered in terms of CPU, this can be seen in the difference in the activation times, which are CPU bound and the similarity of the Import phase which is I/O bound.

In terms of overall runtime, the table below summarises the main phases as I have them recorded

As you can see from the table above, we blew our budget on time massively - which was concerning for everyone and we had a lot of explaining to do, but we had captured a lot of good data as to why we had issues, what we did to resolve them and how we can mitigate them for TC2.

This is shown (as usual) in the table below

As I have said above, we blew the transaction log several times during this Unicode conversion and it is important that I explain why this happened. When running a Unicode conversion, the exports are usually fine, as they are read operations - imports are write operations (DML), these will be captured in the transactions logs/online logs unless they are noted to be part of a Bulk load (Oracle, DB2, SQL Server) and there are SAP Notes to enable you to do this. Similary if you are deleting the records of an entire table, you use the Truncate table command, which is non-logged as well - so far so good.

Truncate table BKPF

The issue with transaction/online logs raises its head when you are deleting records because a Unicode import process, for a table you have split, has failed and you have restarted it. This uses this type of command

Delete from BKPF

WHERE (
"GJAHR" <= '1009' and "BELNR" < '006919999' and "BUKRS" = 'XX01' and "MANDT" = '100'
)

This is a logged operation, and depending on the size of the record set - it could blow your transaction log, but it is unlikely to do this on it's own, more likley as a group of repeated packages performing deletes.

The final issue we had was not being smart about how we restarted all the failed Unicode processes, because we did not manually restart failed processes until the end of each servers Unicode run, when we did restart them we had over 50 Unicode processes all trying to delete from the DB at the same time - as shown above this is an easy way to blow your Transaction/Online log, it caused us a great deal of pain and contributed to the database crash, but we learnt a great deal from it and we applied those lessons to TC2.

So we finally made it through TC1, collected a wealth of important data (which we'll analyse in the next post) and utimately re-affirmed we were on the right path. Next up was Trial Cutover 2, all we had to do was make through Christmas alive!

Running a Proof of Concept - it's really a good idea (CUUC Series Part 3)

Chris Kernaghan — Mon, 04 Jul 2011 01:40:26 GMT

I am great believer in Proof of Concept (PoC) tests, as you can see here, they provide solid proof of an idea and often additional benefits in areas not previously thought about. In the case of the Upgrade and Unicode conversion from my previous post, there were a great many assumptions and things being attempted for the 1st time - so the need for a PoC was evident.

We defined a set of objectives from the PoC to ensure we got the data that we needed, this is detailed in the table below

One thing I firmly believe in is that asking for help is a strength and not a weakness, in this case my team and I were about to run a Unicode conversion on databases which were multi-terabyte and we needed this process to run quickly and smoothly. So from the beginning of the project, I ensured that there was budget to have assistance from Microsoft, HP and SAP if their consulting services were required. When beginning the PoC, I spoke to my friends in the Microsoft/SAP Competency centre about how best to enhance the Unicode process, they were as usual very helpful and technically brilliant. Microsoft provided an ABAP tool called EasyMig, which enhances the table and package splitting abilities of the Unicode process, it does this by using more targetted Where statements within the table splitting routines.

Example screenshots of Easymig Tool

Example of EasyMig statement

tab: ACCTIT
WHERE (
"POSNR" <= '0000000003' and "AWORG" = '2009' and "AWREF" = '4918931532' and "AWTYP" = 'MKPF' and "MANDT" = '100'
or
"AWORG" < '2009' and "AWREF" = '4918931532' and "AWTYP" = 'MKPF' and "MANDT" = '100'
or
"AWREF" < '4918931532' and "AWTYP" = 'MKPF' and "MANDT" = '100'
or
"AWTYP" < 'MKPF' and "MANDT" = '100'
or
"MANDT" < '100'
)

Example of standard R3ta split

Where ("AWREF" < '4918931532')

The certification status of this tool is in question, as it uses a different R3Load than the standard SAP one due to restrictions on Where statement length, but for the purpose of the PoC we decided it was too good an opportunity to miss and we needed to see how exactly it functioned.

The PoC took a lot of effort, there was much work involved in providing the hardware and the right data set to accomplish our aims before we even got started on the Upgrade and Unicode process. Once we got the hardware and the systems in place, we then had to start work on the preparation of the Upgrade and Unicode conversion - it is important to read the notes associated with your upgrade and Unicode process as these are the fixes to the gotchas you will find. This process takes a few days, during which point you can have the client standing over you expecting things to be moving - it is important to be firm and explain the process and the consquences of not following that process.

Once we got the preparation out of the way and got down to some proper work, we hit a number of challenges, which were interesting to say the least as you can see from the table below.

The PoC was a great success and provided answers to each of the objectives, some positively and negated others, but ultimately there were more positive affirmations than negative - which is always a good result.

From the PoC we moved into the DEV and QAS Upgrades and Unicode conversions, which were done on VMWare environments - these posed their own challenges but I will follow this post with the 1st trial run instead.

The unintended consequences of compromise how to not screw up later by fixing something now (CUUC Series Part 2)

Chris Kernaghan — Sun, 26 Jun 2011 11:16:04 GMT

During projects we are all faced with problems and have to find solutions to them, these solutions can range from the very simple to the complex. Quite often though these solutions require some form of compromise, which at the time can be acceptable but later turn into a millstone.

During a recent project I was involved in, the project had many challenges around provision of servers to perform Proof of Concept (PoC) upgrades.
The client was trying to run the project at breakneck speed and the hosting partner was not able keep up, although in my experience this is a function of hosting partners and not any particular individual. Because of this difficulty in providing servers within the hosting partner, the client decided to utilize their internal capability, which on the face of it was a perfect solution. This blog will explain why this was not the solution it appeared to be and why governance around this decision is critical.

The client was undergoing an ECC 6 Upgrade and Unicode conversion from 4.7, they had been on Intel Itanium processors - therefore using IA64 chipsets and binaries. The new environment they were moving to within the hosting partner's data centre was based on the new Intel Nahalem processor - using the X64 chipsets and binaries. As I have blogged before, having a PoC at the beginning of a project is vital, as it validates a number of things which are vital to moving the project into the development phases. But what happens when you are not able to run your PoC in the way you intend to run your later phases.

Obviously it is heavily dependant upon what factors are limiting your ability to run the process you want to. In my case it was a combination of kit availability, processing power in the alternative solution, processor architecture differences.

The table below details these differences more fully

The table above was part of the output in defining our objectives and how these differences would affect our ability to use the results from the PoC.
This analysis yielded the table below, which shows the objective of the PoC, the challenge, the analysis of how useful the outcome would be and the importance of the objective.

The analysis was used to ensure that the solutions we were using did not invalidate the PoC, this subsequent analysis yielded the table below. Using the table the project was able to show how it would mitigate the risks and issues of not executing the exact process planned for Production and also the effect it had later in the project.

As shown above, most of our objectives could be met, but items specific to timings were impossible to validate properly. As a result the project team were not able to properly validate them until the 1st Trial cutover when we were working with Production data volumes and Production hardware.

This ambiguity caused several headaches to the project,

1. We were not able to give the business a full breakdown of the timings to justify our downtime requirement

2. The Steering committee has to 'trust' the project team that downtime had been properly extrapolated

3. The project team had to keep the faith the whole way through the project that we could bring the process in within the agreed window

During projects, clear governance is essential when making decisions   like these - it protects people and the project, it should prevent   issues descending into a 'blame culture' when previous decisions have   negative consequences later.
The project used Sharepoint as a way of   providing public access to the Risk and Issue logs, it also instituted a   Decision log as a way of keeping track of the multitude of decisions   made during the project.Using tools like Sharepoint are a very easy way   to provide structure to certain project functions. For this project,   using the decision log to record items like1. The stakeholders involved   in the decision
2. The decision required
3. What the options are
4. What the decision taken was or the next action to be taken in order   to reach a decisionSharepoint enabled a transparency that cannot be   achieved with spreadsheets, when updating the decision in the log, the   previously defined stakeholders are notified of any changes to a   document that they are linked to. This enables them to check and verify   any information relating to their work.

Project governance, in my opinion, not fun or sexy, but it is essential to the smooth running of a project. I will also put my own hand up as being a poor practitioner of project governance - better than I used to be for sure, but that's by learning the hard way.

Architecting an SAP Upgrade and Unicode conversion (CUUC Series Part 1)

Chris Kernaghan — Tue, 21 Jun 2011 08:02:31 GMT

I blogged last year about a programme of work to run several upgrades and Unicode conversions on multi-terabyte SAP Systems for a client. Finally I have been able to pull together a series of posts which will detail the work, the challenges and the solutions we implemented during the programme. The upgrade programme lasted for 10 months and covered a number of projects,

The upgrade projects (Listed below)
A Data Centre migration and support migration to an outsourcing supplier (Run by the client and the outsourcing supplier)
Hardware migration (As part of the DC move)

These blogs will focus on the upgrade of the ECC system, for the simple reason that it was the one I was closest to, although I will talk about the other upgrades and the challenges they posed. At the start of these projects it is important to have strong idea on how to execute the upgrade and Unicode process, I use the word idea because at this point it is an idea; it has not been validated by anything other than experience from other projects. I looked at the overall programme of work being executed at the client and the client's roadmap for the future and developed an end-point of where the systems should end up at the end of the project - this is detailed in the table below

Once I had decided on the versions of the O/S, the database and the SPS levels of the applications, the main consideration is how do you get to those versions. The SPS levels are quite easy as they can be integrated into the upgrades and handled during the upgrade runtime. The O/S and DB versions changes are usually handled outside the upgrade process, but they can be interwoven with Unicode conversions depending on the scenario and so it is important to look at what else is going on within the landscape to see what options are available to you to achieve your aims - for me my saving grace was to be the data centre migration.

The diagrams below show two options available to achieve the O/S and DB versions, using the Unicode conversion and another doing it seperately.

(Notes: The colour of the servers is important as it reflects a change of hardware
IA64 – Itanium Processor architecture
X64 – X86 64Bit Processor architecture
NUC – NonUnicode
UC - Unicode)

Below is a table which details the agruements for and against each approach and you can quite clearly see a winner

There is one factor in all the of the discussions about options above that I have not dealt with, and that is the database size. Database size is a major driver as to which method of conversion you undertake, the larger the database and the longer it takes to convert - that is a fact. Not all databases are created equal, different vendors, different versions, different levels of administration all play their part in the Export/Import timings, but still data volume is the kicker - in this project a 4TB database had already defined the process as parallel as there was not a big enough window to perform the Unicode conversion as a serial task, the question now was just how much parallelisation did I need to use.

The next post will deal with the actual PoC process and what we found out running it.

My thoughts on the Road ahead for SAP Consultant from a technical point of view

Chris Kernaghan — Sun, 20 Mar 2011 06:22:34 GMT

A few days ago I read from Vijay Vijayasankar and Jon Reed (including Leonardo De Araujo)a few posts about what they saw Road Ahead for SAP Consultants - 2011 in SAP looking like, the trends and technologies that would drive SAP forward. When I read the list initially I found nothing substantive about the underlying technologies that would underpin these whizzy and fancy portents. So I posted on twitter to both of these people that I, as NetWeaver Technical Consultant, was unhappy at being left out of the party. It was a reaction, they to their credit responded very nicely and confirmed they had inferred the technical requirement in many places but for lack of time they could not cover all of it.

So I decided to write this post to hopefully compliment their previous posts, offering a technical point of view to their vision.

1. BusinessObjects reporting tools will get significant traction in BW shops

I have not seen a great deal of detail about how the architecture of the BO tools can and should be mapped into an existing landscape. I have seen many pretty pictures and can infer much from these, but it feels a little subjective and open to interpretation. Also I have been talking to some of the hardware vendors and they are sketchy on some of the detailed sizing requirements. So I see a huge learning curve for the BO 4.0 release and many painful lessons for technical people.

4. Learn what is possible with HANA and what is not

Completely agree with this one, HANA is developing at a furious rate with Chinese whispers about version 2.0 functionality being discussed like it is gospel. Lets bring it back to earth here, first I love the concept of John Appleby’s BFS technically known as an In-Memory database. Second I do not love the sheer lack of technical information on it, there are no SAP Architecture documents about it on Marketplace, no Master guides about it, there are 212 SAP Support Notes for it but no entry in the Product Lifecycle or Installation guides about it. I am not hugely concerned for me, I work for a consulting organisation with strategic partnerships with SAP and the appliance vendors, so I can get at least 3 different views on HANA and how to integrate it into a landscape, but customers’ technical people must be engaged on how this fits with everything.

6. Virtualization will continue to gain traction

After sitting through so many vendor presentations, and analysing the increase in SAPs ratings of servers – I really did being to wonder how customers were going to get a decent ROI without virtualisation, which was not helped by SAP dragging their feet on some platforms. There are several types of virtualisation that are now supported for SAP, VMWare ESX, IBM p Series LPars, HP Integrity VMs, Sun Solaris Zones, Microsoft HyperV, if you add to that the multitude of developing Cloud Services like HP, IBM, Microsoft Azure and the mature(ish) Amazon. You now have a massive amount of choice in platforms, Enterprise architects foaming at the mouth to consolidate every piece of software ever developed and Vendors scrambling to sell the client licenses for a Virtualisation stack that locks them in. Good dialog is necessary and many workshops to help thrash out a workable solution. As regards Cloud, there is too much hype around this, it is at the top of the Gartner Hype Curve and we all know what that means. Technical teams must be allowed to critically evaluate the platform in terms of management, integration, supportability in order to ensure they are able to maintain a stable progressive platform. The business need to challenge the technical team to help them grow!

7. Security and Compliance is a big deal

I hate doing Security and Compliance, but I recognise why it is needed and the quality it can enforce. There are many gotchas in this area around architecting a valid solution – security and technical people need to start talking more, not dictate to each other.

8. Data federation can add significant business value

I want to see system owners opening up their data sets to the Data federator, I can imagine that a major part of the issue is that the technical people do not want others digging in their ‘vegetable patch’ – who knows what they might find.

9. Wait and watch for Business by Design

This is something that has me excited and I have been that way since I was at the Innovation Weekend in Las Vegas Oct 2010, Kai van de Loo explained his vision about ‘De-perimeterising’ the SAP landscape. It resonated with me particularly because I was working on deploying SAP within AWS, and taking advantage of the ability for people to build new composite applications without having to ask Security. This meant that they could derive business value quickly and easily, then go to Security with two things, a working application they can test (not a HLD document) and a solid business reason/backing.

SAP are developing/releasing platforms at a furious rate, SAP River, Streamworks, BYD, OnDemand HANA, developers and business people are being sold on the value of them, of which there is plenty. Technical people are not being shown the architectural designs behind them or in front of them, we again are forced to infer architectural details from diagrams. When developers sell these ideas to the business, they are not able to talk to the security people as to how these services should be consumed in a secure way – they ask me to do it!

This is only going to get more and more common for 3 reasons

1. People are becoming more at ease with using ‘Cloud based’ services, applications like Dropbox, Evernote are more common – this is feeding into the Enterprise and as a result people are more receptive to them.

2. It is easier to sell customers/architects Platforms as they are easier to see and conceptualise than web services

3. SAP will certify Cloud IaaS providers which will make the ability to link with these services much more realistic without the risk of compromising the core network.

I think in the main the guys got it right in both of their pieces, I just misread the intention of their pieces a little. For me right now I think this is an amazingly exciting time to be working in Technology for a number of reasons

1. Technology keeps advancing, and I have to keep learning about some many new things.

2. The connected nature of the SAP community is driving so much right now, and SAP are listening.

As a result of these things, I am able to communicate with many people, link many pieces of information to come up with solutions at really help customers and drive things forward. I probably am able to talk more ‘languages’ than a UN translator in terms of the technology I can converse about – this can often steer me away from incorrect assumptions and trouble, not everyone takes the time to learn about how their area integrates with other technology stacks.

The thing that worries me right now, is that these technologies are moving at such rates, without complete communication from the designers through to the consumers. Fast moving platforms have a habit of moving themselves and their customers either into trees, dead-ends or if they are lucky onto the highway. What I want to see is more engagement with technical people at the front end, not just architects, with roadmaps decreasing to 6 months to 1 year in places – it becomes difficult to plan and architect the right agility into a solution.

People have to stop being so parochial about their platforms and system – yes your system is special, yes it is different from mine, but if we do not talk, and design a good solution, we’ll get told what the solution is.

By the way, we have 14 hours to save the earth!!

The other side of SAP mobile apps - iPhone prospectives

Chris Kernaghan — Sun, 13 Mar 2011 11:00:51 GMT

Since the Sybase acquisition by SAP, everyone has been harping on about mobile apps - it would be getting a little boring if it wasn't exciting.

I have to admit to being quite jealous of all those Sales people and management types who were getting all their nice wizzey dashboards and reports on their mobile device. Being a Basis person I thought there must be tools to help me keep these backend systems up - so I went looking.

Being an iPhone user I will deal only with iPhone apps right now, that is not because I am blind to all but Apple - but because I do not have an Android platform to use, so in the interests of fairness I encourage Android users to dig into their developer Ecosystem and find similar tools to give this post some balance :-)

The tools I found come in several different flavours

1. Information gathering - iPhone apps which I use to find information on SAP when on the move.

iERP Consultant - a very useful application for finding Transaction codes, Tables, BADIs (Screenshot)
SAP Notes Search - search for SAP Notes by number, the next revision will hopefully have a free text search (Screenshot)
MyHelp - an app which taps directly into SAP Help website (Screenshot)

2. VPN technology

Junos Pulse - the iPhone app for the Juniper VPN product
Cisco VPN - the iPhone has a built in Cisco VPN ability

3. SAP access, Web and WebGui - iPhone apps which I use to provide access to SAP systems or platforms

SAPPLAPP - Uses a HTML Gui connection to provide transactional access to the SAP system - I love this app, it has transformed how I support systems. Just stick Web Dispatcher in the DMZ and you have access to your full landscape using something like this

Using the Web Dispatcher, you can connect to an non-standard port and be forwarded into the Solution Manager system using the Management VLAN - from here it's possible to RFC into all the systems in the landscape. (Screenshot)
StreamWorks - Does exactly what it says on the tin, gives you access to SAP Streamworks. I have personally used this on my last project and found it to be very useful, especially when keeping decisions up to date. (Screenshot)

4. Alerting - iPhone apps which can be used to monitor alerts coming from the SAP system

Redwood Cronacle - Provides access to the CPS system so you can check your scheduled jobs
BizBox Direct - A workflow approval system which looks very interesting and useful
SAP Request Management System - A remote transport management system which pushes notifications of transport workflow items

5. Amazon Web Services (AWS) Apps - iPhone apps which I use to provide complete management of AWS landscapes and Servers

AWS Manager - This is an amazing app, which gives complete access to the EC2 API, from this app it is possible to create and manage an entire AWS landscape (Screenshot 1, Screenshot 2)
Mocha RDP Lite - As most of my demo environments are Windows based, I use this app to connect to the Windows Desktop and manage the server (Screenshot)

These tools have enabled me to do some amazing things, for example I was able to build an AWS Cloud based SAP system whilst being driven up the M5 by a colleague - granted the iPhone screen was a little small for the job but it did work in a pinch. This is not an exhaustive list, just a selection of the tools I have used directly for SAP, there are many other useful tools which I use to free me from my desk :-)

I am always on the look out for new SAP iPhone tools to help ease my life in SAP, and I want to thank all the great inventive iPhone developers for making these tools and helping me in my job. If there are any new tools released or in Beta, tools that you think deserve a mention or Android tools that need talked about - leave a comment to discuss.

Consolidating the learnings from our harsh lessons in MDMP Unicode conversions.

Chris Kernaghan — Tue, 13 Jul 2010 07:24:42 GMT

During my last blog post, I talked about how my previous project started an MDMP Unicode conversion. At this point in the story, we had completed two Unicode conversions, but without much success on converting the data correctly.
With the help of Nils from SAP, the project team sat down and took a long hard look at the problem, as well as where our pain points lay. We had enough data to establish that our big pain points were Financial accounting tables for the Russian codepage issues, also the client had some custom tables which were pain points.
The project team took a number of decisions from this analysis
1. Another person needed to be responsible for the Unicode conversion language assignment, because I was becoming a Single Point of Failure - I understood the process better than anyone else on the team. This also reduced the pressure of two heavily technical roles to a single role. (We actually ended up with two people, which I have to say is better.)

2. We would not attempt any archiving or data migration steps to try and correct the data, because we knew where we had problems. If we had done any data migrations, we ran the risk of changing our known problems into unknown problems.

3.Setting only English as ambiguous

4. Executing two more dry runs

5. Using SUMG to repair data, not the reprocessing logs from SPUM4, this was because the Reprocessing logs were not as controllable as SUMG.

We had set the stage for another attempt, this time we were confident we could get it right. So we ran the CUUC process, and ended up with a system that was actually quite useable :-)
There was a lot of stuff to fix in SUMG, but we had accomplished a Unicode conversion and most of the data was intact.
This was for three main reasons
1. I stopped trying to do too much at once, during the CUUC process I executed all the technical steps. Also I stopped trying to optimise the Unicode process for speed, we were much more concerned with data quality - so we decided to take the hit on the performance.
2. We executed a lot of work on the vocabulary assignment for unknown words and also we delved deeply into the Reprocessing scans. Using the reports um4_analyse_replog and um4_replog_stats, myself and the language team worked hard to resolve collisions and unknown words.

It is very important to note here that quite often this process does not yield a correct assignment for a word, especially with related languages like Polish and Russian. The best that someone can hope for is to achieve the least wrong answer, which does not create too many reprocessing logs.

3. Better use of table comparison tools to automate comparison of data from pre and post conversion extracts

Once we delivered the 3rd system, the response from the business and the project team was one of greater confidence, we now had a process that worked and we knew the data well enough to go for our final dry run before Production.

We set out very clear criteria for our final dry run, it had to meet strict conversion completeness standards as well as run to time without major incident.
The process started with a copy of Production, upon which we executed a final round of language assignment, again bringing the total of unique words below 10000. We then tackled the reprocessing logs, here we did not want tables with over 100000 entries in them as this would produce too many reprocessing logs for SUMG to handle.

Next we ran the CUUC process just like we would on Production, which completed successfully. Following extensive testing, it was found that a few of the tables had failed their conversion threshold - but the team was confident that these could be fixed in Production.

At this point we were flying high and the Steering committee had given permission to start preparations for PRD, which I will address in my next blog post.

Starting another Unicode migration, this time the challenge is 4TB worth.

Chris Kernaghan — Mon, 05 Jul 2010 09:07:18 GMT

As many of you will know I am no stranger to the Upgrade and Unicode conversion (CUUC) process, having completed many of the them over the last ten years, my most recent being an MDMP conversion.

This time I have accepted a project which will stretch my technical expertise again, the data volumes involved are substantial - the client's Production R/3 database is 4TB in size, the BW system is over 2TB, the SRM system needs a two step upgrade process and the SCM system is split into three (Optimiser, Livecache, SCM application)

Thankfully I have a great team working with me, most of whom I have worked with previously on another CUUC project. So I have few worries about people being up to the challenge, and we all work in similar ways, which means we are all able to support each other easily and effectively.

The first challenge I had to work through was simply, how was the team going to work through such massive data volumes, when we had an effective window of 2.5days to accomplish our respective CUUC's. Some people will be screaming at their monitors with answers :-), but really the only sensible answer is to perform a parallel export and import. This is where the database is exported and written to a shared file system, once each package has completed it's file that file is made available to the import process - which then picks it up and starts importing it.

As you can imagine, the hardware requirements for such a project are extensive and so this is not a decision which can be taken lightly. So I decided to go out to some people for some 2nd opinions, I contacted technical account managers in SAP, HP (hardware vendor) and Microsoft (O/S & RDBMS vendor), everyone I spoke to advised the same thing - in order to shift that amount of data you must use parallel export and import.

During the bid process for this project, we found that the client was also planning to undergo a data centre migration, which suited me perfectly. The data centre migration, whilst it can complicate some things, also relieves pressure in one very key area - hardware contention.

Now things are falling into place, I have a method to execute the CUUCs and I have a project to help support my seemingly insatiable need for servers during this process. During my next few posts I will detail the various challenges and decisions of the project.

Consolidating the learnings from our harsh lessons in MDMP Unicode conversions.

Chris Kernaghan — Mon, 05 Jul 2010 09:06:17 GMT

This post offers some advice to people embarking upon their Unicode process, it is definitely not an exhaustive list of things to do.

Your company or client has decided to do either an Upgrade and Unicode conversion or just a straight Unicode conversion of your MDMP System, the first piece of advice I will give you, my technical colleagues, is be prepared for a wild ride and to be saddled with many responsibilities.

An MDMP system is a system which serves many different countries where the languages cannot be displayed using the default SAP 1100 code page, as a result different codepages were introduced to expand the number or characters applications could support. Unicode is capable of displaying every character in every language, so it simplifies many of the system operations.

Due to the data intensive nature of the process it is necessary to have a cross discipline team to be responsible for this part of the process.
Having been through this process, I would define my project dream team as having the following members
1. Basis consultant - responsible for the running of the Unicode conversion process
2. Data migration consultant - responsible for ensuring that the language scans and processes are done properly and vocabularies are properly maintained. (this could also be an internationalisation expert (I18N), but there are not many around)
3. A client representative - responsible for talking to the business to determine the data flow of processes, as well as how the tables and data is being used
4. Language assignment team - responsible for assigning the unknown words in the vocabulary to a language and a codepage.
5. Good team lead with a strong technical background.

This is a wish list, and is only based on a single project, but I probably got much heavily involved than many basis consultants. Effectively I took on the roles of 1,2 and 5, from the list above.

I am going to break down the timeline for our Unicode conversion process by each system.
First we ran the process against DEV, this was challenging because 4.6C does not support the full pre-Unicode tools, there is no SPUMG or UCCHECK - instead there is a limited tool called SPUM4 which is used to scan every word in every record in every table to ensure that it has a language assigned to it. If a word in a record is detected as not being present in a vocabulary, then it is flagged as needing assignment.

We engaged SAP and received assistance from one of their I18N experts, it would be fair to say he wrote the book on the SAP Unicode process. With Nils we ran the scans throughout the system, and started the data analysis of the results. We found a very nasty surprise within the vocabulary - users from Russian had been entering data in a non-standard way, the users had been using the I18N settings as shown below. This meant that the data in the system was effectively using Microsoft Russian codepage ASCII values, not SAP codepage ASCII values. This meant that if a word was not assigned correctly between the codepages, the ASCII value of the letters will be wrongly converted and the word will be corrupted.

After much deliberation, we established that the data within the DEV system was not great, and we needed to know the scale of the problem, so we completed the language assignment to within 10000 unknown unique words and ran the CUUC process. Once the conversion was completed we found massive levels of corruption in the database, far too much to fix using SUMG.
We learnt a valuable lesson about the Russian data entry, that it was going to be one of the major challenges throughout the process. We also decided to use copies of Production to improve our data quality and provide a more iterative approach to the vocabulary conversion.

The project team obtained a copy of Production and copied it back to create a new QAS system. We began the process anew, but this time we did three new things,

Introduced a new codepage, this codepage is designed to accommodate the Microsoft Russian words (English/Russian)

Repurposed a language using the same codepage as SAP Russian (codepage 1500), in this case BG - this became our SAP Russian (Russian/Russian)

Add both SAP Russian (RU) and English (EN) to the ambiguous language list. This means any word the system recognises as being either language is placed in the vocabulary to be checked.

Once we got within our comfort zone of 10000 unique words, we executed the CUUC with worse results than the previous run. This was because a table UMGCCTL (the Unicode control table) became corrupted during process and meant that all the tables were converted using codepage 1100 as the R3load process could not determine the correct codepage for each record. This was a horrible turn of events as the technical team gave up much of their Christmas to complete the CUUC process, but there was a silver lining.
We had another items to check to ensure a correctly running export and conversion. It also prompted the project to grant another 2 attempts at conversion, and also a repeat of the Unicode process on QAS. This time it was completed successfully, but we had chosen the wrong road when assigning English and Russian/Russian as ambiguous.

At this point the project team were a little bruised and battered, but we had learnt a great deal about the process, these lessons would give us a great deal of confidence in the later phases, because you can learn more from your mistakes than you can from your successes.

What value a Proof of Concept

Chris Kernaghan — Mon, 28 Jun 2010 00:33:55 GMT

As I wrote in a previous post, I was not able to run a PoC of an MDMP Unicode Migration. So I want to explore the value of a PoC and find out how other Technical consultants view it.

Just to be clear about what I mean when I say PoC of a Technical Upgrade project, it is a copy of Production less than 6 months old which is in a similar environment to Production. These points are important as the PoC, in my view, has to accomplish several aims.

1. Provide a clear indication of what functionality the upgrade will 'break' after the upgrade has finished.

This is hugely important as it allows your developers and functional consultants a 1st time view of what work needs to be done.

By giving them this view, they can plan any functional changes easily, for example if the client has a customised business process which is now SAP standard. The functional consultant can plan a revert to standard.

2. To highlight areas of concern for the technical team.

Because the technical team are providing most of the heavy lifting during the Go-Live weekend - we have to be aware of where the problems lie. The PoC provides a clear view as to where these problems are, we can then either fix them so they do not re-occur or we can fix them in place and then plan for the same error in future executions.

3. To validate the scope of the project.

Sometimes the client has not fully taken into account how the upgrade will affect their Enterprise Architecture when they have gone out to tender. By validating the core business processes with the business and the functional consultants, the project team can validate or invalidate many of the assumptions made by both parties during the tender process.

Similarly sometimes the client has picked the wrong thing to do, either they are addressing their issue in the wrong way - perhaps an upgrade is not the right thing to do, it might be better to re-implement.

4. Provide the client with a sense of momentum.

Projects can be slow to start sometimes, and seeing a technical team sourcing and providing systems straight away gives the client a feeling that things are happening, rather than watching a lot of functional consultants attend workshops were the output will not be tested for weeks. (I am not dismissing the valuable work my functional colleagues do in anyway.)

5. Make the client see a customised upgrade process from the first button press.

It is easy to spend the first 2 to 3 weeks reading every SAP note and manual, having a massive checklist of things to go through, but this is wasteful of a consultant's time and the client's money. By using a PoC we are able to execute a reasonably 'vanilla' upgrade and begin customising an upgrade process from the first button press. The first few days are spent reading the major SAP notes and manuals for the very obvious issues and applying experience of previous upgrades to the process. This will yield a reasonable list of actions, then we can start the upgrade - where we will hit more issues than we have in our list. These issues will be fixed and added to the process, which will be refiend with every further upgrade - until the client has a fully customised upgrade process.

But what about the reasons for not doing a PoC upgrade, I have not heard of many but here is a selection of them and why I think they are poor reasons.

1. There is no time to do the PoC

As we have said above the PoC allows the whole project team to see how the upgraded version sits with both customer data and business processes, by doing this critical decisions can be made without affecting the to-be landscape.

2. The budget does not allow for it

It is my experience that not having a PoC often leads to additional costs for two reasons -

A. Because architectural decisions have be made in DEV, often this means it needs refreshed post-project to bring it in line with the eventual decisions made in the project.

B. Processes have not had time to integrate within the project team during upgrade execution, so often there is a requirement to run a second dry-run before go-live.

3. We do not have enough hardware for the PoC

SAP mandate that there needs to be a minimum of 2 systems in order to support an SAP application, so in order to run an Upgrade project there needs to be more hardware available, so plan accordingly.

Often an upgrade project utilises new hardware to take advantage of both higher performance (often required for the new application) and the ability to reduce the SAP hardware estate.

Even if a migration to new kit is not involved in the project, there should be an expectation set early in the process that an upgrade is a hardware intensive process.

So if anyone has any views similar or perhaps different to my own, leave a comment so we can discuss.

chris.kernaghan

SAP Music Playlists - an experiment

Landscape Virtualisation Manager concerns

What can you do to keep up with TechEd Las Vegas when you cannot be there

Trial Cutover 2 (TC2) completed and the lessons we learnt from it (CUUC Part 6)

Data analysis of CUUC outcomes of TC1 (CUUC Series part 5)

The 1st Trial Cutover (CUUC Series part 4)

Running a Proof of Concept - it's really a good idea (CUUC Series Part 3)

The unintended consequences of compromise  how to not screw up later by fixing something now (CUUC Series Part 2)

Architecting an SAP Upgrade and Unicode conversion (CUUC Series Part 1)

My thoughts on the Road ahead for SAP Consultant from a technical point of view

The other side of SAP mobile apps - iPhone prospectives

Consolidating the learnings from our harsh lessons in MDMP Unicode conversions.

Starting another Unicode migration, this time the challenge is 4TB worth.

Consolidating the learnings from our harsh lessons in MDMP Unicode conversions.

What value a Proof of Concept

The unintended consequences of compromise how to not screw up later by fixing something now (CUUC Series Part 2)

My thoughts on the Road ahead for SAP Consultant from a technical point of view