There are many pitfalls to beware of when upgrading your NetWeaver landscape (by installing a new SP or EHP). The preparations and logistics themselves can be exhausting, not to mention the pounding heart when you restart the server for the first time. Let's assume everything went smooth, all those weeks of preparations paid off and you can sit back and relax (hey, did you ever have doubt?).
A few minutes later, the phone rings - an end user is on the line saying that something in the portal is not working. You find out that there is a javascript error. You ask the user to clear his browser cache and try again - BINGO! Success. Back to the chair. The phone rings again, same issue different user. Now you realize the magnitude of the problem - it is possible that some small js file was changed and the incompatibility with the old version in the browser cache is killing your success story. Is it possible to automatically clear the cache for all users in the company? But what about all those with laptops that connects from home? And the manager that is now in an important meeting abroad?
Well, you get the picture.

What could you do to avoid this?
Prior to the upgrade you could change the cache configuration in the server from 1 week (the most common configuration) to 1 day, and later reduce it to 2 hours. This will make sure that the old resources would be kept in the browser cache for a minimal time so you will likely to avoid this problem. This introduces additional tasks and worries - did you remember to make the change in the configuration? Did you remember to set it back to the original value after the upgrade? Is there a performance impact on your server? Did end users feel this?
Not the best solution, but unfortunately you didn't have better option.

Introducing the "resource versioning" solution
A new feature in the portal allows you to gain better control of your static resources.
As you know the html pages in the portal are mostly dynamic, when they are built on the server side there is an API for embedding the javascript, css, images and other resources within them - it is usually done by using PRT's IPortalComponentRequest getResource(...) method (the recommended way). Raising a flag in the portal configuration will make the PRT add a unique string to the end of each resource URL, this string is unique to the application that contains the resource and being regenerated on each deployment of that application.

The result will be a URL that looks similar to this:

http://... Irj/portal/... /someScript.js? 1c506cf475c726e63cf9a72279a06f3f

Your code will look similar to this:

IResource myResource = request.getResource(IResource.SCRIPT, "scripts/ someScript.js ");
response.include(request, myResource);

The browser will store this script in the cache under the full URL, including the unique string after the ‘?'.
Each time you deploy an application the unique value will be regenerated, so if this application contains static resources and they will be requested again by the PRT API the link will be created with the new identifier and the cached version will not be used.
Since you usually don't make frequent changes in production there is no risk to performance (because of invalidated cached content). However, if you have an emergency fix, or an upgrade, you can safely assure that end users will get a fresh copy when they connect to the portal.

Not all resources in the portal are using this versioning feature, but the good news are that those who don't usually have similar mechanism. In order to benefit from this new feature in your custom apps make sure you use this API for adding your resources to the response. Now you get this control for your custom applications for free, could it be any better?

Are there any special considerations?
One side effect is that the browser may hold redundant resources in cache, i.e. two scripts with same name but different identifiers. The browser cache has a policy for managing its storage and should quickly overcome this, when the cache threshold is reached the old resources will be deleted. No worries here.
Is it expensive to generate this identifier? The identifier is generated in the deploy process anyway so there is no additional overhead for generating it - just remember that each time you deploy an application the identifier is regenerated, even if you deploy the same application with absolutely no changes.
One more thing, if at runtime you ask for the resource without the identifier (ask for http://.../someScript.js) you will get the current version and it will be kept in cache under this name. Same with any random string you put there (ask for someScript.js?abcd) , there is no mechanism on the server side that validates this string to match the application identifier - it is just for controlling the version on the browser cache. So, if you run load test or have automated scripts that call resources you don't have to use complex correlation rules to match the identifier - just be sure to clear the cache first.

A thought - if all applications in the portal used this method then theoretically you could cache resources forever, knowing that when a resource is changed you have full control to replace it immediately for all users. Unfortunately this is usually not the case - the cache expiration configuration in NW is applied on all resources served by the server weather they have version id or not, and some do not have it so they might get "stuck" in the cache without a way to control them.

How does it look like?

Before activating the versioning feature:

After activating the versioning feature:

How do I activate it?
Login to the portal as System Administrator, navigate to System Administration -> System Configuration, and on the detailed navigation choose Service Configuration. Browse the Portal Runtime tree -> right click Central Configuration, choose "Configure". Look for "js.SupportVersion" (I know, not the best name) - change the value to "true" if you want to activate the feature, "false" will deactivate it. Perform a restart and you are set.

It is easily reversible and doesn't make permanent changes (besides the resources on the browser cache), so don't be afraid to evaluate and play with it.
Suggestions, feedback, thoughts and constructive criticism are always welcomed :-).

Availability?
The feature is available in the following versions:

• NW 7.0 EHP2 SP0 - the default value is "false" 
• NW 7.0 EHP2 SP2 - the default value is "true"

And an important update
my colleague Irit Okshtein collected and compiled a "Performance Best Practice Guide for NW04s", it includes important practical and conceptual information that can really help you out. If you have NetWeaver landscape you should read it!
The guide can be found in the following link:

Best Practices - Portal

There comes a day in someone's life when he has to use a network sniffer to make sure all those bits and bytes on your super-server are sailing in the right direction. Cases like general network slowdown, strange HTTPS behavior and other "funny" phenomena's may indicate that your Net is not up to the work anymore.

When you use such a sniffer, a whole new world is appearing - no, it's not the matrix, but close enough. A sniffer like WireShark can show you what happens under the hood and enable you to identify DNS resolving issues, proxies, SSL handshake problems and other network errors, as well as network related configuration problems. One of the problems that usually come in mind is involved with the TCP/IP packet checksum check,  if you have a bad line it may be that some of the data is being corrupted and thus leading to bad checksum value - the packet needs to be retransmitted and you suffer from the delay.

What is Checksum for TCP/IP?
in short - when you send TCP/IP packets you need to make sure the data was not altered by mistake due to network errors, the method used is CRC (Cyclic Redundancy Check). CRC takes the message stream as input and produces as output a calculated short representation of the message. The value is added to the TCP/IP message and later on someone on the receiving end can run the same method on the message and compare the two - if they are the same, most chances are that the message was not altered. Not the same? This packet is corrupted and will have to be sent again.
For TCP/IP packets a Checksum algorithm is used, which is rather simple and effective. The checksum is calculated and added to the IP and TCP headers.

What does it have to do with performance?
Checksum calculations take CPU cycles. Since the server CPU is also in charge of handling the TCP/IP stack and creating the messages it is "wasting" resources on these actions, when instead your could squeeze more "juice" for your application.
Here come the network adapters to the rescue - network adapters manufacturers embedded functionality in the network adapter itself so it can perform several actions in the hardware instead of consuming server CPU for that, all you have to do is to tell the server that you "offload" some of its responsibilities to the adapter.

What can you offload?
There are multiple operations but I think the most important ones are:

• TCP checksum offload - the process of calculating the packets checksum is delegated to the adapter.
• Segmentation offload (LSO - large send offload) - large packets are divided into smaller frames for better efficiency. This option means that the adapter will handle the segmentation instead of the server.

There are few more options but you will have to explore it yourself, you won't believe how many articles and studies you can find on these two alone...

In Win OS you can find it in your Network Connection -> Properties -> Configure -> Advanced

as shown below...

Is it really good?
The million dollar question! There is some debate about this, it depends on your network, hardware and application usage profiles but you should be aware of it because in some adapters this is enabled by default and for others not. For example, see note #1023047 for AIX settings recommendation.
In theory if you have a fast CPU which is not running at full capacity it may be faster to use the CPU to calculate the checksum, using the adapter may give you less data throuput. But this is a big discussion and the manufacturers/providers have different recommendations according to your hardware (for instance - IBM and HP...).
My recommendation is to experience with it first and test the impact on your landscape, the numbers I found shows about 0%-15% changes in CPU, but again - it depends on so many parameters that it is not possible to extrapolate.

Why is it so important to know that?
Closing the circle that we started with, when you use a network sniffer on a system that enables TCP checksum offloading you may be mistaken to think you have a network problem. The sniffer is "sniffing" the network packets before they arrive to the network adapter, so the checksum value in this stage will be incorrect (we didn't try to calculate it yet). So you have a scary error message in the logs "TCP checksum incorrect" or something similar (depending on your software). So don't panic, and before you dig your wiring from the ground check your adapter settings first.

That's it for today, came out longer than I thought. Hope I didn't get you to sleep during working hours. For me, it made me feel much more respect for the people who work nights and days to make it possible for us all to connect.

<body><p> </p><p>Reminder - If you are reading this post before reading my previous ones I kindly ask you to go back and read the first ones first... I'm trying to lead you through a process. Thanks :)) </p><p>So by now everything looks a little brighter - the browser is set, you have a browser tracing tool and you have seen how things looks like from your end. Hopefully (or not...) you spotted some problems and fixed them already. Never thought there is so much action behind the scenes? We are just getting started. </p><p>Today we will dive deeper into the http traces and learn some more about the client-server dialog, as well as some tuning tips on the server side that might save you some time.</p>h3. Into the deep water

<p>The http spec defines many different header names that can be used, most of them were defined in http spec version 1.0 but some are only introduced in http 1.1 spec. To add to the confusion, some of the headers are explicit (meaning you specifically define what behavior you want) and some of them have implicit meaning (you get some behavior as result of other header you defined). Add to the equation the browser version factor and you get a nice cocktail that will certainly give you a headache. You think you are bigger than that? You will be knocked down when proxy servers joins the party. Don't tempt me. </p><p>Unfortunately (and luckily for you) I won't be able to cover all headers or different combinations, the spec is unbelievably long and far from being easy to read or understand. I will refer to the common scenarios and naïve interpretation of the spec, I know there are always exceptions and special cases - want to share one? this is why we have the "comments" section at the bottom. <br />Let's take an example and analyze it.</p><p>You know by now how to take HTTP traces, I will continue to use HTTPWatch for the demonstration - to see the headers of the request and response you should click the URL and navigate to the "Headers" tab. See image below.</p><p> </p><p><img  />//weblogs.sdn.sap.com/weblogs/images/59389/httpwatch_headers_tab.JPG|height=384|alt=httpwatch headers tab|width=598|src=https://weblogs.sdn.sap.com/weblogs/images/59389/httpwatch_headers_tab.JPG|border=1!</p><p> </p><p>Usually the headers are separated by line breaks and looks like <header name>: <header value> but in this tab they are displayed more conveniently (for raw data go to "Stream" tab).<br />The spec defines which values are possible for every header name.</p><p> So let's review the more important headers.</p><p>Consider the below headers, this is the Login post:

POST /irj/servlet/prt/portal/prteventname/Navigate/../.. HTTP/1.1

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, /

Referer: http://localhost:53000/irj/portal?...

Accept-Language: en-us

Content-Type: application/x-www-form-urlencoded

UA-CPU: x86

Accept-Encoding: gzip, deflate

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)

Host: localhost:53000

Content-Length: 298

Connection: Keep-Alive

Cache-Control: no-cache

Cookie: PortalAlias=portal; saplb_*=(ilperf125_F37_30)303540750; ...

POST /irj/servlet/prt/portal/prteventname/Navigate/../.. HTTP/1.1<br />First line says POST or GET, which is the http method we use, following by the requested URL and then HTTP/Referer: http://localhost:53000/irj/portal  (http://localhost:53000/irj/portal)?...<br />Who triggered this request; this is useful in case you want to trace which file caused this request to be sent.

Accept-Encoding: gzip, deflate<br />the browser explicitly specifies that it support gzip data compression, so the server can compress the response body. Just to clear things out - a) it is up to the server to decide if it wants to compress or not, b) the http headers themselves can't be compressed; you need the headers to tell the browser that the body is actually compressed.

Cache-Control: no-cache<br />in this case the browser requests that the request-response will not be cached. This is normal for POST since it involves with submitting parameters that needs to be processed on the server.

Connection: Keep-Alive<br />this means that the browser will reuse the connection after it receives the response, unless the response specifies otherwise (like Connection: close).

+Cookie: PortalAlias=portal; saplb_=(ilperf125_F37_30)303540750; ...

+On every request the browser sends the complete list of cookies that are relevant to that domain. Remember I said that the headers can't be compressed? This is a major downside for cookies since they can be very long (and could have been easily compressed since they are textual).</p><p>And now let's go the server response (headers received by the browser):*

HTTP/1.1 200 OK

Server: SAP J2EE Engine/7.00

Content-Type: text/html; charset=UTF-8

Content-Language: en-US

Content-Encoding: gzip

Date: Mon, 21 Jan 2008 15:29:24 GMT

Transfer-Encoding: chunked

HTTP/1.1 200 OK<br />The server responds* with the response code (talked about it last time) and the http version it supports.</p><p>Content-Type: text/html; charset=UTF-8<br />the content type of the returned body, In our case this is html file. </p><p>Content-Encoding: gzip<br />the content is compressed. In httpwatch under the "stream" tab you can see the raw request and response. If you look below the headers you will see gibberish - this is the actual compressed content (binary data). Go to the "content" tab and you will see the body after it was decompressed.</p><p>Date: Mon, 21 Jan 2008 15:29:24 GMT

The date and time this message was originated.</p><p>Transfer-Encoding: chunked<br />the server does not provide total response length.

</p><p>Let's see another example - this is a response of resources that should be cached:*

HTTP/1.1 200 OK

Server: SAP J2EE Engine/7.00

Content-Type: application/x-javascript

Last-Modified: Tue, 01 Jan 2008 09:40:19 GMT

Cache-Control: max-age=86400

Content-Length: 69729

Date: Wed, 23 Jan 2008 15:44:59 GMT

What is the difference here?<br />You can see the content type header that indicates this is a javascript file, and the cache control returns with a "max-age=86400" value. This means that the server tells the browser it should cache it for 86,400 seconds, which are 24 hours. From now on, for the next 24 hours the browser can load this file from cache whenever it is requested with no need to request it from the server again.

What happens when the 24 hours are over?

The server sent an additional header "Last-Modified", this is the date when the requested resource was last modified on the server. The browser saves this date together with the cache definitions of the file and when the cache period times-out it sends a conditional request to the server asking if the resource was modified or not. If the resource was modified the server will send the updated resource file and replace the cached version, if not - the browser will send a short response that indicates the browser to continue using the cached file (and continue to cache it for the next 24 hours).

What can you do to improve performance?

The http server allows you to configure many parameters, you should check what is configured and make sure to have the best matching configuration for your needs.

How?

Use the VisualAdmin tool. Go to Server_abc -> services -> HTTP Provider service

On the Runtime tab you see the following 4 options, make sure they are properly configured.

*

So you think the server is tuned, you Get your free I(E)Tunes and the Chop your logs!, but something still doesn't flow right... what can it be? How can you trace it? Do you think the server response is optimized? Let's be sure of that!

Finding performance problems doesn't always mean you need to take your administrator permissions and dive into the bowels of the monster (a.k.a your portal server). You can get a pretty good picture of what is going on using your browser and some http tracing software. This week I want to give you a glance of some tools that can help you trace the problem from the client side. Next time I will demonstrate how to find these pesky annoying problems, identify them and off course - take care of them once and for all.

You will be amazed of the amount of information you can reveal from this easy-to-do client side tracing.

What tool are you looking for?

You are looking for a tool that will help you monitor the client-server chit-chats, easy to do if you work for one of the intelligence agencies... if not, you will need a gauze soaked with anesthesia, pair of scissors, lock picking equipment and some heavy tool to beat the network's administrator password out of him. OR just keep on reading :)).
The best tool will be one that is plugged into the browser and can show what is loaded from cache and what is going over the wire, second in best is a "proxy like" tool that can only show what is going via the net, it will also require additional configuration. On this point I will say that I don't have any specific preference for one of the tools, SAP has a license for one so I use it and it gives me what I need, I don't see any clear advantage in one of the other tools I mention below. If you have different experience the "comments" part below is craving for content...

For IE we have the following tools (partial list of tools):

HTTPWatch - http://www.httpwatch.com/

HTTP Analyzer - http://www.ieinspector.com/

Debug Bar - http://www.debugbar.com/

For FireFox

http://www.getfirebug.com/

Proxy tools (won't show you the cache requests)

http://www.fiddlertool.com/

How to work with it?

In each tool it is slightly different but the principal is the same, you click some "start" button, perform your scenario (for example, login to the portal) and then "stop". You can now analyze the data that was collected. What you are interested in is the request headers, amount of content, whether or not the response came from local cache or server, what was the response code etc...

Screenshot of HTTPWatch trace of the portal:

Marked with red you can see the http requests and some basic info about each one, below are the details (marked in blue).

Next time we will discuss real-life examples of client-server communication.

Until next time,

Yoni.

If there is a first thing to check on a production server it must be the log configuration.

You will be amazed to know how just a few logs in INFO severity can cause your server to slow down (besides filling-up your valuable disk space). In the portal code there are many calls for writing log messages in different locations, it allows you to instantly collect info when you are in a problem. BUT, the appropriate severity setting for a production system should be "ERROR". If you don't have specific issue there is no need for the server to collect data for nothing. What's more - you can change the severity on the fly at any given time.

The way to change the severity levels is via the AdminTool, under server -> services -> Log Configurator. Make sure  that "ERROR" is defined for both "Categories" and "Locations" tabs and use the "Copy severity to subtree" button (See below). This is important since there may be some "INFO" hiding in lower levels. Don't forget to save your change and make sure you use the "apply to all server nodes" option.

I ran a 3 hours test to get some numbers for you, my base line was navigations test where all logs are in "ERROR" and then I just set the "com.sap.portal.prt" location to "INFO". Before reading next, care to guess the results? 

The CPU level for the server jumped by 20% (absolute number), response times were doubled, number of full GC events doubled and the log files were hundreds of MB is size... makes you think, isn't it?

Some techi stuff - The log method calls are creating complex String objects causing memory consumption and file system I/O when writing them to disk. They are designed to be completely ignored if the corresponding log severity is not active, so they won't add overhead to the server.

If you do run into a problem and need to change the log severity, try the following:

1. Activate only the locations you need - setting everything to INFO will collect huge amounts of data and make it very hard to get a clear picture from the logs.

2. If you want to reproduce the problem on production environment try to do it on an isolated server node (off the load balancer), this way you ensure that all collected data is relevant to the scenario you are running and no other users will be affected.

3. Document your changes and time-limit them - don't forget to set the log severity back to ERROR when you are done.

Until next time,

Yoni.