SAP, Itanium, Linux and virtualization ...

[Former Member]

If you've been there, these keywords without Linux usually lead to HP-UX and HP VSE OE with Integrity VM. But, if you try to find legitimate and practical alternative, it turns out that RHEL with Xen is the only good choice.

I have several system copies running on RHEL5.4 Linux with Xen virtualization platform, most of them are SAP ERP based with Oracle 10.2.0.4, ECC 6.0 EhP3, both on HVM Windows Server 2003 guest systems and on PV RHEL5.4 guest systems, few IDES systems, one with NW7.0 and Portal. All this is running on HP Integrity / Itanium 2 hardware (BL870c), and FC SAN, HP EVA 5000 and EVA 8400 is used for storage. What I'm interested here above all is to hear if there is anybody out there having some similar story (SAP, Itanium, and virtualization) ? Please, speak out if you are listening ...

I have some documents prepared and in preparation with details (I will put it here or in blog if there is interest), but for now just in short: systems are quite good, performance could be compared with some physical systems and production servers, but above all it is very stable. We also use VMWare for x86/x64 in our company, so I am able to compare these two platforms. There are many pros and cons, but I think that both offer good solutions. One of the main conclusions is that Xen on Itanium compared to HP-UX offers similar if not even better performance (that is, at least until HP finally gets implemented AVIO for HVM Windows guests), and if you are not in need of some high-end features (including live migration), Xen is better. There were some bumps on the road, though - Oracle showed me a bug (some workaround is available, though), but the biggest of them all is that RHEL decided to abandon support for Itanium in RHEL 6 ... SLES is a good option, but it is not supported by SAP at the moment. Any comments, questions, hints, insights, advice ?

ZP.

[hannes_kuehnemund]

Zoran,

... but the biggest of them all is that RHEL decided to abandon support for Itanium in RHEL 6 ... SLES is a good option, but it is not supported by SAP at the moment.

SLES is supported by SAP. Check SAP note [171356|https://service.sap.com/sap/support/notes/171356].

Regards,

Hannes

[Former Member]

Yes, I know that and I'm glad it's so, but have in mind four keywords in the subject of this thread as topic - what about Xen ? In 962334 - Linux: SAP on Xen virtual machine it is stated "Caution: On the IA64/Itanium Platform, only Red Hat Enterprise Linux 5 U1 and later is supported.", which is a pity, though SLES was one of the options I was testing in my prototype environment. If you have any information about this in future or anything related, that would be useful.

ZP.

[hannes_kuehnemund]

The reason why SAP does not certifiy SLES Xen on IA64 is, it simply does not exist

[markus_doehr2]

> The reason why SAP does not certifiy SLES Xen on IA64 is, it simply does not exist

Oh - I also wasn't aware of that - interesting.

Giving the fact that RHEL 6 does no more support IA64 and this, the question is, if Itanic is still a strategical architecture for non-HPUX shops...

Markus

[Former Member]

I've heard that it's so, I wasn't sure - ok, I could compile my own Xen kernels for Suse, but SAP support wouldn't like that ... and I know there is a Xen branch for 2.6.27/29 kernel supported by Suse (correct me if I'm wrong about something), but it's probably not for IA64 ? I can only conclude that in the future of SAP on Unix/Linux-ia64 after Red Hat's decision to abandon Itanium on RH6, I'm left only with HP-UX (unless Novell decides to do something more). HP-UX is an excellent OE, but it's price demanding and more suited for mission critical environments and similar - if I compare Xen with HP Integrity VM from the standpoint of performance and reliability, HP Integrity VM doesn't offer more than Xen, if any at all - it is more about availability and high-end reliability which most business environments don't need. It's a shame Red Hat is not willing to poke HP and compete somehow - I'm sure they have some economic picture, but I'm not sure how accurate it is: based on number of currently sold Itanium servers - are they really dead, I didn't hear anything from other vendors ?!!? I only know that 60% of Itanium servers sold today is Unix (mostly HP-UX), 30% Linux (mostly Red Hat), 5% are Windows ... and the "latest" news is that Oracle joined the Xen advisory board: http://community.citrix.com/display/ocb/2008/12/18/TheOraclecomestoXen ...

[Former Member]

I meant high-end manageability instead of high-end reliability for Itanium VM ... I know that it was a known fact (e.g. roadmaps 7 years ago or so by HP and IBM), and EPIC struggle against RISC ) or for it wouldn't know now ... and one just for the history:

http://www.itaniumsolutions.org/resources/red_hat_enterprise_linux_virtualization_and_intel_itanium_... ...

[markus_doehr2]

> It's a shame Red Hat is not willing to poke HP and compete somehow - I'm sure they have some economic picture, but I'm not sure how accurate it is: based on number of currently sold Itanium servers - are they really dead, I didn't hear anything from other vendors ?!!?

All other vendors (Bull, Unisys, Dell, IBM...) abandoned IA64 because it's too expensive (too few numbers of them) and too slow; the cost/performance ration is miserable, the x86 32bit compatibility mode is much slower than expected. You have to spend a lot of money for an e. g. 4-way box; for the same money you can buy an even more powerful x86_64 system or two of them.

> I only know that 60% of Itanium servers sold today is Unix (mostly HP-UX), 30% Linux (mostly Red Hat), 5% are Windows ... and the "latest" news is that Oracle joined the Xen advisory board: http://community.citrix.com/display/ocb/2008/12/18/TheOraclecomestoXen ...

Itanium is due to its design best suited for technical/scientifical (floating point) processing, not that much for integer processing (what SAP basically does - including the databases) or for running virtual machines (ABAP/Java). Because of the nature of it being an EPIC processor all the optimization logic must be put into the compiler. I guess, the open source community will not be able (just due to the lack of manpower and knowledge) to make gcc as good as the HP-UX aCC compilers.

We have and had several Linux/IA64 boxes which were replaced by x86_64 (mainly AMD). For half of the price you get at least the double throughput; IMO there's no more justification for preferring iA64 over x86_64 any more.

Just my EUR 0.02.

Markus

[Former Member]

>

> All other vendors (Bull, Unisys, Dell, IBM...) abandoned IA64 because it's too expensive (too few numbers of them)  and too slow; the cost/performance ration is miserable, the x86 32bit compatibility mode is much slower than expected. You have to spend a lot of money for an e. g. 4-way box; for the same money you can buy an even more powerful x86_64 system or two of them. 
> Markus

I ment, not just hardware vendors (Oracle, Microsoft, SAP, ...) - IBM had the most performant system based on SAPS and benchmarks for long time (AFAIK, if it didn't change in the mean time, Oracle also made a competition, I don't know if anyone took the claim), but it had even worse cost/performance ratio than HP Integrity. If you want high end performance, Itanium can give you solid single-threaded horse power that x86 couldn't offer (again, AFAIK), and this is very important for databases and some business systems - it is not always about parallelization and number of concurrent users per system.

>

> Itanium is due to its design best suited for technical/scientifical (floating point) processing, not that much for integer processing (what SAP basically does - including the databases) or for running virtual machines (ABAP/Java).  Because of the nature of it being an EPIC processor all the optimization logic must be put into the compiler. I guess, the open source community will not be able (just due to the lack of manpower and knowledge) to make gcc as good as the HP-UX aCC compilers.
> 
> We have and had several Linux/IA64 boxes which were replaced by x86_64 (mainly AMD). For half of the price you get at least the double throughput; IMO there's no more justification for preferring iA64 over x86_64 any more. 
> Markus

Unfortunately or not, my management decided to move on with Itanium (but unfortunately for sure they decided years ago to go into production with Windows). You are right up to some point - Itanium is EPIC, but it is not just for scientific and FP, why would they've been sold successfully as platform for database servers for years ? There are IA64 CPUs with additional FPU and cache, and there are different versions and platforms - what did you exactly compare ? Generally it's still not that clear IMO. And here comes my favourite anegdote about Itanium raw performance: while doing an Oracle 10g patch (CPUJan2009 bundle, I think or something like that), carelessly for some reason (I usually don't do it that way, of course) I haven't read the patch README and I have omitted to do additional view compiling by habit (many patches/patchsets do not need it beside the usual compiling). Production database node has two dual 1.1GHz processor modules and 8 or 16G of RAM at that time (mx2 modules, let's say it is like 2 CPUs with dual cores), and the moment I have had database ready for use I have concurrently read that I need additional compiling which needs about 30 minutes for average database with 2000 objects by internal Oracle benchmarking/testing. I have sighed, did query and saw three to four times more objects in my database than stated in the README manual, and started swearing in myself about complicated corporate procedures and flows about informing users, system availability, internal QA, etc. So, I decided to put system back down with my teeth strongly clenched together and started these two scripts on production system (not the usual situation, I repeat). It lasted not more than 5 minutes ! I didn't have to follow all the strict QA procedures, inform users, and so on ...

[markus_doehr2]

> Unfortunately or not, my management decided to move on with Itanium (but unfortunately for sure they decided years ago to go into production with Windows). You are right up to some point - Itanium is EPIC, but it is not just for scientific and FP, why would they've been sold successfully as platform for database servers for years ?

Because customers that were running PA-RISC need a roadmap and and a successor CPU and since an "emulator" (ARIES) was already available... why not take that path

> There are IA64 CPUs with additional FPU and cache, and there are different versions and platforms - what did you exactly compare ?

I compared a 4-way IA64 with 32 GB (rx4640) with a 4-way DL585 with 32 GB RAM on SLES 10 running identical programs on the identical database version (MaxDB) with the identical setup (RAW devices).

I'm not saying it doesn't work, I just say that the alleged performance you get does not compare to the added money you have to spend.

Markus

[Former Member]

>

> Excuse me for my badly formatted reply, I have now corrected that:

>

>

> > It's a shame Red Hat is not willing to poke HP and compete somehow - I'm sure they have some economic picture, but I'm not sure how accurate it is: based on number of currently sold Itanium servers - are they really dead, I didn't hear anything from other vendors ?!!?

>

> All other vendors (Bull, Unisys, Dell, IBM...) abandoned IA64 because it's too expensive (too few numbers of them) and too slow; the cost/performance ration is miserable, the x86 32bit compatibility mode is much slower than expected. You have to spend a lot of money for an e. g. 4-way box; for the same money you can buy an even more powerful x86_64 system or two of them.

I ment, not just hardware vendors (Oracle, Microsoft, SAP, ...) - IBM had the most performant system based on SAPS and benchmarks for long time (AFAIK, if it didn't change in the mean time, Oracle also made a competition, I don't know if anyone took the claim), but it had even worse cost/performance ratio than HP Integrity. If you want high end performance, Itanium can give you solid single-threaded horse power that x86 couldn't offer (again, AFAIK), and this is very important for databases and some business systems - it is not always about parallelization and number of concurrent users per system.

>

> > I only know that 60% of Itanium servers sold today is Unix (mostly HP-UX), 30% Linux (mostly Red Hat), 5% are Windows ... and the "latest" news is that Oracle joined the Xen advisory board: http://community.citrix.com/display/ocb/2008/12/18/TheOraclecomestoXen ...

>

> Itanium is due to its design best suited for technical/scientifical (floating point) processing, not that much for integer processing (what SAP basically does - including the databases) or for running virtual machines (ABAP/Java). Because of the nature of it being an EPIC processor all the optimization logic must be put into the compiler. I guess, the open source community will not be able (just due to the lack of manpower and knowledge) to make gcc as good as the HP-UX aCC compilers.

>

>

> .. to be continued ...

[Former Member]

>

> .. continued ...
> 
> We have and had several Linux/IA64 boxes which were replaced by x86_64 (mainly AMD). For half of the price you get at least the double throughput; IMO there's no more justification for preferring iA64 over x86_64 any more.
> 
> Just my EUR 0.02.
> 
> 
> 
> Markus

Unfortunately or not, my management decided to move on with Itanium and hardware migration is not an option now (but unfortunately for sure they decided years ago to go into production with Windows). You are right up to some point - Itanium is EPIC, but it is not just for scientific and FP, why would they've been sold successfully as platform for database servers for years ? There are IA64 CPUs with additional FPU and cache, and there are different versions and platforms - what did you exactly compare ? Generally it's still not that clear IMO.

And here comes my favourite anecdote about Itanium raw performance: while doing an Oracle 10g patch (CPUJan2009 bundle, I think or something like that), carelessly for some reason (I usually don't do it that way, of course) I haven't read the patch README and I have omitted to do additional view compiling by habit (many patches/patchsets do not need it beside the usual compiling). Production database node has two dual 1.1GHz processor modules and 8 or 16G of RAM at that time (mx2 modules, let's say it is like 2 CPUs with dual cores), and the moment I have had database ready for use I have concurrently read that I need additional compiling which needs about 30 minutes for average database with 2000 objects by internal Oracle benchmarking/testing. I have sighed, did query and saw three to four times more objects in my database than stated in the README manual, and started swearing in myself about complicated corporate procedures and flows about informing users, system availability, internal QA, etc. So, I decided to put system back down with my teeth strongly clenched together and started these two scripts on production system (not the usual situation, I repeat). It lasted not more than 5 minutes ! I didn't have to follow all the strict QA procedures, inform users, and so on ...

[hannes_kuehnemund]

There is a document from an SAP consulting/migration company (RealTech) available.

It is called: [The Trend from Unix to Linux|http://www.realtech.com/wInternational/kontakt/contact-REALTECH-Novell-whitepaperW3DnavanchorW2621100414310042.php]

Just to let you know, what RealTech (actively in the migration market) sees at customer site (losing/winning paltforms)

Thanks,

Hannes

[Former Member]

>

> ...
> I compared a 4-way IA64 with 32 GB (rx4640) with a 4-way DL585 with 32 GB RAM on SLES 10 running identical programs on the identical database version (MaxDB) with the identical setup (RAW devices).
> 
> I'm not saying it doesn't work, I just say that the alleged performance you get does not compare to the added money you have to spend.
> 
> Markus

Let me put it this way (not to go into details about production technologies for both architectures) - we have production on old rx4640 with Madison cores (about which I told the story, no HT, no VT-i, mx2 modules, 1.1GHz), but we also have newer rx4640 with Montecito cores, and Integrity BL860c and BL870c blades with Montvale cores (1.6GHz, HT and VT-i enabled) which showed to be roughly three times faster for some critical long running SAP jobs and transactions. I can tell that it scales up very good, and that one 2p BL860c used to cost around 3000 euros, while similar x86 Xeon based Proliant or blade costed 2000 euros - it is not that big difference, but the difference in raw power is immense. That is why I asked what is being compared exactly. I had also to measure difference between old CPUs on production-like rx4640 and same machines with new Montecito cores on 1.6GHz, and real life experience is at least twice faster jobs and transactions, while the price of CPU upgrade is questionable - maybe almost half the price of new Integrity servers to replace these, or same as x86 based replacement (but not worse than that), and that is still less by far then to migrate whole landscape to new CPU architecture (but that is not the main reason for the decision to stay with Itanium). And I must admin that I still find any Integrity more robust than eny Proliant ...

[Former Member]

>

> There is a document from an SAP consulting/migration company (RealTech) available.
> 
> It is called: [The Trend from Unix to Linux|http://www.realtech.com/wInternational/kontakt/contact-REALTECH-Novell-whitepaperW3DnavanchorW2621100414310042.php]
> Just to let you know, what RealTech (actively in the migration market) sees at customer site (losing/winning paltforms)
> 
> Thanks,
>   Hannes

I will go through it, thank you.

ZP.

[markus_doehr2]

> And I must admin that I still find any Integrity more robust than eny Proliant ...

Well, don't know what experiences you made but this is what I see on one of our x86_64 boxes (and it's not the only one):

$ uptime
  1:38pm  up 1272 day(s), 4 min(s),  2 users,  load average: 5.14, 8.09, 9.09

and yes, this is only an academical thing of course

If you try the recent new Nehalem x86 CPUs you may be surprised what those tiny little machines are able to handle - in sense of raw single threaded CPU power; ask HP for their SAPS sheet and compare the IA64 to x86.

I'm not saying it's a wrong choice. I just notice that all big vendors pull away from Itanium (Oracle on Windows 2008 is not there, DB2 9.7 is no more there on Windows and Linux), all the major projects planned (Montecito) were cancelled (quite long ago), Sun wanted to do once an IA64 port which they abandonned, IBM stopped selling IA64... all those things led us switch to x86_64. Since Linux on IA64 is LittleEndian it wasn't more than a backup/restore.

We also still have those old boxes (rx4640 and rx2600) and we still use them for small systems, the boxes run and run and run, no doubt.

Markus

[Former Member]

I don't know about the others, but Oracle usually just makes delays in patch and major releases for Itanium (as did previously for Alpha and maybe some other architectures as well) ... I will check that and SAPS about Nehalem, though - it might be interesting, thank you for your thoughts.

ZP.

[Former Member]

Just one or two more ... reason I have started this thread is not that much about Itanium's future (though HP should think over some things, it is not just about putting additional cores on EPIC CPUs, community is more important), but about Red Hat's opportunity to poke HP backed up by Xen and Open Source community for IA64 (it's still there !) ... I still think it's a hasty decision, to make RHEV with management server based on Windows application, and then to opt out Itanic (is the situation that bad ? I didn't see it that way, nobody sent that kind of economical signals like Sun did many times in the past). I know, Sun's got SPARC, IBM's got POWER ....

[markus_doehr2]

> Just one or two more ... reason I have started this thread is not that much about Itanium's future (though HP should think over some things, it is not just about putting additional cores on EPIC CPUs, community is more important), but about Red Hat's opportunity to poke HP backed up by Xen and Open Source community for IA64 (it's still there !) ...

Yes, I understood that; what I was trying is to find out the reason for that - which may be just because of the low market share of IA64 on non-HP-UX operating systems. It's been a niche and it will stay like that.

Markus

[Former Member]

>

> There is a document from an SAP consulting/migration company (RealTech) available.
> 
> It is called: [The Trend from Unix to Linux|http://www.realtech.com/wInternational/kontakt/contact-REALTECH-Novell-whitepaperW3DnavanchorW2621100414310042.php]
> Just to let you know, what RealTech (actively in the migration market) sees at customer site (losing/winning paltforms)

I went quite carefully through that document, but I will most likely come back to it later several times if needed - it is insightful, but expressing fact through figures is usually a good way to show a clear picture, but it can also make things dim. Main objection about making any conclusion based on this document is about scalability and other conceptual principles which are not visible there for different reasons. Above all, not just about some grid or cloud computing trend, most business systems (SMB) are using scale out philosophy, not old mainframe-like scaling up - most x86_64/x86 systems are also about this. If you look closely, "Correction 1" in that document is related to BL860c exactly, a 2p server with 5850 SAPS (as it can be seen also on http://www.sap.com/solutions/benchmark/sd2tier.epx), and this gives not more than 0.85 EUR/SAPS which is comparable to given Proliant systems. Of course, 0.37 ratio for x86_64 systems is far better, that is perfectly sound and I am not against conclusion coming from that fact (there are also other important cost saving factors), but 19.91 EUR/SAPS with Superdome is quite unreasonable for most users - all the features most big SAP systems could ever need are available with blade servers and under 0.85 ratio (how big demands should it be if you involve 4p BL870c and scaling out with Netweaver and Oracle RAC ?) ... such benchmarks are still rare, including those with virtualization, but nevertheless, I would avoid drawing wrong conclusions. BTW, Sun SPARC shows marvelous SAPS/CPU ratio (better than any other, as I see there) ... so, I don't see quite simple loosing / winning platforms there ...

[Former Member]

>

>... what I was trying is to find out the reason for that - which may be just because of the low market share of IA64 on non-HP-UX operating systems. It's been a niche and it will stay like that.

I guess you are right ... inspired by all this talk, I've decided to put a blog about results I have so far: /people/zoran.popovic2/blog/2010/01/27/sap-linux-virtualization-and--itanium ... so, go ahead and shoot ...

ZP.

[Former Member]

BTW, I've briefly browsed through the TPC-C and TPC-H results, and there are some interesting things to see - TPC-C is comparable to SAPS, and you can make wrong conclusions just looking at top 10 or wahtever: http://www.tpc.org/tpcc/results/tpcc_perf_results.asp?resulttype=noncluster&version=5%&currencyID=0 ... there are not many respectable price/tpmC results ... but TPC-H shows a bigger space for conclusions - based on different database sizes there is a big variety of results: http://www.tpc.org/tpch/results/tpch_results.asp?orderby=hardware

Now, what I am trying to say is that the power of scale-out approach (e.g. having clustered systems) is underestimated (in general, not just about this example): HP BladeSystem c-Class 128P RAC (http://www.tpc.org/tpch/results/tpch_result_detail.asp?id=109060301) has higher score than most, and a good price/performance result - yet, it is not an Itanium blade, but you can get the idea from an expensive Superdome (http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=107022701) ... there are really small number of business cases which could justify buying a mainframe or something like that (couldn't imagine easily), unless some extreme mission critical requirements are more important (Itanium was generally built with that in mind, and Intel's income coming from it is almost the same as from x86_64, so it will be there for quite some time). I suppose we are going to see very soon some results with blade systems, and that would might be a good occasion to continue the discussion ... unless vendors like SAP and Red Hat don't get acquired by IBM in the mean time ...

ZP.