zoran.popovic2

January 2010 Previous month Next month

 

h1. *Introduction*


 

I have made this article based mainly on proposal I have created last summer after finished tests on prototype systems (detailed document available on my LinkedIn profile), and additional tests and unofficial benchmarking data will follow later in new posts. This text is is mainly intended for those who own SAP on Itanium and Windows, and/or for those interested in SAP, Itanium and virtualization. Many things happened in the mean time, one of which is the Red Hat's decision (by the end of last December) to discontinue support for Itanium (or IPF, Intel's Itanium Processor Family) in the announced future release RHEL 6. This was not so unexpected, and RH states that this decision is made because of the small number of Itanium systems sold in the recent period, and which doesn't tend to grow. But this story is not likely to end that fast (RH is fully supporting Itanium on RHEL 5 to the end of it's life in March 2014, while giving extended support until year 2017 through OEMs), Oracle bought Sun who has SPARC architecture and other hardware (there was a failed attempt to port Solaris to Itanium), enters the Xen advisory board (http://xenbits.xensource.com) with people involved in Open Source contributions, Oracle Enterprise Linux and Xen IA64 (Itanium) ports. RH6 is to be supported on IBMs POWER architecture, and so are other "traditional" RISC/CISC CPU architectures, x86 and x86_64 (http://kbase.redhat.com/faq/docs/DOC-23403). There are no clear indications from other software and hardware vendors about Itanium's future (at least not the ones I deal with here having their support: HP, SAP and Microsoft), but there are current benchmarking results and research studies with predictions.

 



Back in 2003 my
company bought HP EVA 5000 storage and OpenVMS cluster with two Alpha
ES47 nodes to support Oracle 9i RAC database for our legacy
application, along with a rack of Xeon based Proliants (it's a kind
of tradition, before them we had Alpha and VAX systems). A year after
or more, it was decided that we should migrate to SAP. It was an
ambitious project: we had 9-12 months from start to going live,
implementing BW3.5, ERP 2004 with MM, WM (having MFC automated
warehouse), FI/CO, SD, PP, QM, PM, TM, logistics and procurement, etc
(maybe I have omitted something), everything that we had in the
legacy Oracle application but HR, CRM and some remote locations
(which were covered in later roll-out and optimization projects). We
also implemented Solution Manager as project base and maintenance
base, having all the necessary strict Q&A and GMP procedures. All
this had to be supported with appropriate hardware and
infrastructure. The first choice was (among others) was HP's offer
with 10 Integrity (Itanium based) servers and HP-UX. Somewhere in the
early beginnings, our project management decided to change OS
platform to - Windows !!! Today, five years later, we have at least
twice larger system (1TB database in size in production, number of
users - up to 800 with expected growth, additional affiliate
production plants, automated warehouses, end users, etc) working
quite good with only minor changes (added RAM up to 16-24G in
production, same number of CPUS: 2x mx2 1.1GHz per node), and some
infrastructure improvements. We never had any serious unplanned
downtime or system failures, performance, reliability, availability
and stability was predictable (apart from some OS problems with
Microsoft MSCS and one short storage outage).

 



But, in the past
year we had a large roll-out and optimization project involving ERP
upgrade from ECC5.0 to ECC6.0, and it showed that project members,
developers, IT staff and key users always needed (and still need)
additional sandbox systems, IDES demo systems, system copies and
similar for different purposes (from experimenting and unofficial
testing, to training and Q&A, project preparation and operation,
etc). This is showing the need for new level of flexibility which
could be only obtained with virtualization. There were many options,
from buying new physcial machines (very expensive even for less
expensive architectures), manual consolidation (MCOS and MCOD, with
or without Microsoft WSRM - Windows System Resource Manager), thin OS
(single kernel) virtualization (like VZ, Parallels Virtuozzo, but as
we have strict security standards and procedures which conflict with
OS patch level for VZ, this wasn't convenient even for testing), to
full virtualization platforms. It turns out that with Itanium you can
only have HP-UX VSE / Itanium VM or RHEL if you want official SAP
support (as explained later). There is also a very interesting and
sophisticated SAP solution - Adaptive Controller, but for now, Xen on
RHEL is doing a very good job. I have two HP BL870c hosts with 6
active sandbox systems (homogeneous copies of ERP development, Q&A
and production, BI/SEM IDES with Java instance, ERP IDES, etc) and
they all work properly and very stable. From pefromance standpoint,
HP-UX with Windows guests doesn't offer more at all (without AVIO
drivers, similar to PV guest drivers on Xen which don't exist only
for Itanium), the only thing I miss at the moment is Live Migration
on RHEL/Xen (which Integrity VM supports, as it also has some other
nice-to-have features). But I am able to move virtual machines to
different hosts manually without a problem (multipathing and shared
storage does it's work).

 



SAP supports
virtual guests, without responsibility about their performance, and
database on production systems should be on physical servers if
maximum performance is needed - otherwise, there aren't important
reasons not to do it even with database. We use Oracle 10g at the
moment (10.2.0.4) on Windows, which works fine on guest systems - but
because of Windows, it has to be HVM, full virtualization. For
optimum performance (and many other good reasons), the best option
would be migration to RHEL, including production systems (all the
tests show that it would be a smooth transition). There is a general
trend about migration of SAP systems from Unix to Linux (for all the
good reasons), while migration from Windows is less popular. Thing is
that 60% of all recently sold Itanium servers are Unix (read: HP-UX),
35% are Linux (RHEL and SLES), and only 5% are Windows
(first link among references).
Microsoft has no intention to introduce Hyper-V on Itanium (as Citrix
also doesn't currently, because the Xen code branch they bought
didn't cover ia64), but it is supported in all important flavours on
Windows Server 2008. Migration to a different hardware platform is
not an option, just as migration to HP-UX at the moment (current
number of Itanium servers does not justify that risk and expense).
But, the old aging hardware should be either upgraded (but it costs –
additional CPU costs almost as one BL860c, and though it's partly
consequence of local vendor's clumsiness, it is not very
reasonable), or replaced with new servers – or, used for
non-production (until dead), which is aligned very good with
virtualization (RHEL/Xen paravirtualization is available for old
Madison cores, but HVM isn't). There are better reasons to migrate to
RHEL beside this (money saving driver) – decision makers can decide
to change CPU architecture at one point, but with RHEL we can already
have the level of flexibility we need – far better than in any
other option: HP-UX is too expensive, and Windows doesn't offer it.
One important aspect is coming from GMP and other compliance issues,
so we need same platform both on test (Q&A) systems and in
production, and this justifies even more migration to RHEL5.

 




So,
the starting point was: SAP ERP and BI systems on Windows Server 2003
on HP Integrity servers and HP EVA storage virtualization solution, and the current stop is: some of those
systems working on Xen fully virtualized guest machines on Windows
Server 2003 and on HP Integrity blade servers, and some working on
paravirtualized guest machines on RHEL5. Next stop would be more
systems involved, and finally – everything migrated to RHEL, either
on Xen virtual machines or bare metal. Also, I find that Integrity blade servers are far more affordable (comparable to Xeon based Proliants) than expensive mid-range cell based andother Integrity flavours.





!https://weblogs.sdn.sap.com/weblogs/images/251885800/rh04.jpg|height=504|alt=image|width=665|src=https://weblogs.sdn.sap.com/weblogs/images/251885800/rh04.jpg|border=0!

 

 

*Current
environment and Itanium platform*




Currently, in the environment I am describing here, IPF (Itanium Processor
Family) architecture is being used exclusively for the needs of SAP
systems in my company. There are 18 such servers at the moment (10х HP rx4640 + 2х
rx2620 with Madison cores, 2х rx640 and 2х BL860c with Itanium
dual-core Montecito cores, 2х BL870c with Montvale cores), beside
additional x86 HP Proliant DL360 G4/5 servers (4x without additional
servers for support SAP routers and Web Dispatchers which are in
Microsoft Cluster for production just as SAP central services on
production and test systems, or on VMWare platform for support and
other purposes). Beside VMWare as main VTI for x86/x86_64 processor
families, HP EVA Storage nodes are used as form of FC SAN Storage
virtualization (also using MSA2012 for VMWare, additional EVA 8400
obtained, but not yet ready for production).

 




SAP landscape currently consists of:


    • BI
      systems (development, test (Q&A) and production), BI7.0 SP18


    • CEN
      system (transport domain controller, CUA, partly central
      monitoring), NW04


    • Solution
      Manager (MOPZ, EWA, monitoring, project roadmaps, ticketing, and
      other)


    • SAP
      routers (for end users, and for support and external access)


    • SAP
      Web Dispatchers (BI portal, WebGUI)


    • network
      printing servers with SAPSprint, SAP Web Console for some RF
      terminals


    • different
      sandbox and other systems: homogeneous system copies, traning
      systems, IDES systems, etc. - all working as Xen guests at the
      moment


h2. *Equipment
has life time (it is aging)
*




Among these 18 servers, 12 servers are entering
5th year of usage: rx4640/2620 servers on which lies the main SAP
landscape (it consists of ERD/Q/P, BWD/Q/P systems). All these
servers are very reliable and stable – there never was any serious
hardware failure or issue on production servers (or even any incident
as a consequence) until this day ! But, with hardware aging, support
becomes more expensive, additional components or spare parts also
become very expensive, and also vendor desupport dates are getting
closer and closer for equipment and different functionalities (usual
practice is to have maintenance renewal periods and IT equipment
amortization within 3-5 years, but sometimes it makes sense to extend
it). There are at least two possible roads: continued use and
planning about Itanium platform, or migration to another processor
architecture (as mentioned). If such migration is planned, then all
other technological aspects and existing options must be taken into
account (some less demanding and ”painful”, some are not, but
offer different advantages and overall total cost – just as
changing the OS platform, changing the server vendor, storage vendor,
etc. should be considered as well). Some kind of trade-in model or
amortization were not practice up to now as they are not offered by
all available vendors currently (used or refurbished equipment is
also not used in production systems in critical business environment like this) and probably it also
will not be in respective future. There is an option to stay on
Itanium platform by sole migration to Itanium blade servers which
has for consequence mandatory need for additional servers based on
current requirements for the main landscape, but without all the
other additional systems (this is not justified and will be explained
further on).

 




*Itanium
servers – unused brute force*





Main argument for Itanium system application is
usage within OLTP systems and databases – an example: for CPU patch
19 for Oracle 10.2.0.4 based on standard README instructions,
downtime lasts about half an hour (given for some internal Oracle
tests for a database with 2000 views and 4000 objects, while our
production system has at least twice as more), on production systems (old 2p mx2 rx4640 1.1GHz) it lasts about 5 minutes. But, except for biggest loads
during the day (peak is usually 13:00-15:00 on work days and in the
end of the month), this power mostly remains unused during the rest
of the day (average CPU load on our production does not go over 10%
on central instances with database and central service nodes, and up
to 35% on dialog instances, having at least 400 active users among
800 users) and this is expected state. Consolidated CPU usage and
other server resources usage can be realized by installing additional
SAP or DB instances on the same OS instance (operating system
instance, or in general OE, Operating Environment) on the same
server, allowed by SAP MCOD or MCOS installations – but, all there
many possible problems of coexistence of such instances during their
work, usage and maintenance life-cycle. That is one of the reasons
why consolidated OS environments should be isolated and that is
usually done through some form of virtualization. It can be justified
above all simply by using non-productive and productive systems and
their total cost – but, completely virtualized production
environments have became a standard experience today and also a need
for many business environments (if it is possible to scale resources
and estimate bottom lines, then it is just necessary to take into
account the “price” for virtualization).







Existing solutions for virtualization which were
considered while preparing this text: Citrix
XenServer, Hyper-V
/ Win2k8, HPUX
VSE/VM, vPars, nPars, Parallels
Virtuozzo / OpenVZ, Novell
Suse / Xen, Oracle
VM, Fedora
/ Xen, FreeBSD
/ Xen, Red
Hat Enterprise Linux (RHEL), and so is Centos / EPEL / xenbits, Sun
Virtual Box, QEMU/KV, VMWae - there
were also other solutions which were not considered for obvious
reasons - some solutions were eliminated from the start
because those are not available for IPF and are not even planned to
be supported and working on Itanium any time near (Hyper-V, VMWare,
XenServer, Virtual Box, and Oracle VM up to some point) but only
x86/x86_64 (I am dealing with SAP environment which is mostly based on IPF platforms and Windows
OS, apart from some remaining OpenVMS and Alpha systems). Following
criteria are applied on remaining solutions:

 




    1. only
      full virtualization (hardware based, HVM) or paravirtualization
      (PVM, more favoured) with hypervisor is supported (solutions which provide full
      isolation) – HP vPars, Virtuozzo, OpenVZ represend OS / “Single Kernel“ virtualization which is not supported (by SAP and other
      vendors) on production systems

    1. all
      other official requirements given by SAP as a software vendor for
      our production systems, and by all other software and hardware vendors about platform support


    1. fully
      working installation prototype which confirms solution feasibility
      in our system environment, e.g. boot from SAN


For HP-UX it is
also mandatory to have additional expenses for HP installation and
maintenance support including additional preparations, training and
similar activities which are not needed for RHEL (though, HP-UX
systems offer high level of availability, manageability and more,
they represent top of the business and industrial standard, but it is
very questionable if this is really needed). RHEL Xen still does not
support many advanced features (like Live Migration, PCI
passthrough, paravirtualized Windows drivers – HP-UX doesn't
support those drivers too, etc) which are easily enabled on other
platforms (x86, x86_64) or which HP-UX supports, not likely that all
of the will be, but some might be.


Furthermore, HP BL870c can support almost 4
completely isolated active guest systems (even on physically
different processors, CPU partitions avoiding kind of context switching – HP nPars can not go
in granularity under the physically available resources, including
CPU, Integrity VM can), and without a significant performance loss it
can support up to 7-15 such guest systems (having 16 HT cores)
compared to 1-core based bare metal systems, and more if high-end
performance and load is not expected. Of course, number of inactive
guest systems is almost unlimited – ready to be started and used
when needed at any time if resources are available, without
additional reinstallation, side-effects and server setup.

 



*Business
requirements*




Business requirements, coming from business case which emerges here during
development, testing and usage of all SAP systems, are putting high
demands about fast changes and mechanisms which enable such
flexibility having all the existing business standards and SAP usage
standards preserved (GMP and validation practice, QA,
availability/reliability SLAs above all, etc). Specific requirements
which arise from these are:




    1. one
      or more system copies in a given time (usually ASAP), homogeneous or
      heterogeneous – as part of the strategy for testing and/or
      development (but not for Q&A needs), and by given requirement or
      expressed need (testing of new or risky features, system
      types/usages and functionalities)

    1. change
      of the existing architecture / SAP Landscape – for instance, the
      problem of the peak loads on the Q system during testing periods
      (systems were never sized for these needs), Support Pipeline (to
      circumvent transport change and validation requirements in order to
      speed up cut-over after system upgrade, roll-out and similar),
      training and sandbox systems – last two examples show in our
      practice that system copies are far more efficient and usable than
      any other solution (aligned with storage split-mirror technologies
      like snapshots and snapclones) compared to unjustified client copies
      or exports on systems with already more than 1 TB in DB size, which
      can not support frequent requests for Q system data refreshments.
      One temporary solution is also to combine those two procedures (as
      used in R/3 system upgrade to ECC6.0 here during year 2009), and even
      more efficient would be to slightly change validation practice
      (which in essence remains same, but using far more efficient system
      copies). This is aligned with SAP note
      432027 - Strategy for using SAP Support Packages, which
      describes additional sandbox and test system usage (e.g. sandbox in
      Upgrade Master Guide) and as part of the official landscape.

      *A
      step further*
      (a solution and a problem) would be involvement of additional
      development and Q&A systems, but also using more efficient
      change management and automated test procedures using SAP Solution
      Manager and eCATT tools which we already have available at hand
      without additional investments (but there also many, many other
      useful tools) except for additional planned effort by validation and
      different SAP teams. Every testing of changes during any of the SAP
      projects is a convenient opportunity for preparation and
      implementation of such tools and solutions.


    1. Changes
      during system patching or system upgrade which require alternate
      system landscapes (as during the former mentioned upgrade in year 2009),

*Solution* - Pool of Servers






The strategy which enables the fastest possible
response on large number of requests: certain number of servers is
prepared like an “on hold“ template (firmware / hardware, FC /
network, SAN storage and other needed resources, then the OS
installation, Windows domain name reservation and AD infrastructure
settings, antivirus / firewall / proxy, account specifications and
all other OS settings, Oracle RDBMS and SAP s/w installation), and
then by a well defined procedure a copy is made using HP Storageworks
Business Copy technology in a shortest time according to SAP
standards (homogeneous copy based on database copy, with additional
post-installation activities which last not more than an hour).
Server in the pool can be activated in a very short time, deactivated
or renewed with fresh data (just by making database copy and
post-installation tasks again).

The
only problem with this solution (as mentioned about the missing
features for Xen on Itanium) is a very poor disk I/O. This is the
consequence of the needed full virtualization (HVM) for Windows
guests (3-6 times slower than on bare metal, while network I/O is
less deteriorated). This makes critical impact on database
performance in some cases (though parsing is done faster than on
older bare metal systems), which makes the whole ERT system in
average slower (e.g. closing the financial period on guest system takes 4
hours in batch instead of 2.5 hours as in production system). There are many workarounds and solutions for this
problem - using paravirtualized I/O drivers if it was possible,
iSCSI/NFS storage approach (not top performance, again), etc - but
the best thing would be moving database to physical RHEL machine, using
MCOD wherever possible. Database nodes wouldn't be easily
reallocated, but one of the possible solutions in future would also
involve adaptive infrastructure (SAP ACC, with Solution Manager -
Adaptive Computing Controller) as part of the SAP strategy for system
usage and consolidation:




    1. all
      SAP systems can adapt to business needs very fast and in a flexible
      manner




    2. dynamic
      load balancing according to system load (physical and virtual)




    3. easier
      SAP landscape maintenance

 

Optimal
solution would involve ACC because not only that it avoids better
possible human errors, but it also follows vendor recommendations and
standards better given with SAP Solution Manager as the base
technical tool which is really useful in all system maintenance and
management tasks according to GMP / ITIL / ITSM, and which is free of
charge (unless additional Service Desk Extended component and
functionalities are needed for non-SAP systems and Service Desk /
Help Desk, and similar). Therefore it makes sense to make additional
effort and improve this system and it's usage in our environment.


 



!https://weblogs.sdn.sap.com/weblogs/images/251885800/vnc.jpg|alt=image|src=https://weblogs.sdn.sap.com/weblogs/images/251885800/vnc.jpg|border=0!


 


*Realization options*



These are the only existing usable IPF
consolidation platforms for pool of servers and their perspectives (having Windows on Itanium systems):




    1. acquisition
      of new physical servers (without hypervisor and virtualization)





Pros:
most efficient (but not optimal) use of existing knowledge and
resources




Cons:
price of 1 Itanium server is between $5К and $20К in average, and
more (without support agreements and other hidden maintenance costs,
human labor)




    1. HP-UX
      Integrity VM hypervisor:




        1. HP
          Integrity VM (software virtualization) as part of VSE (Virtual
          Server Environment) offers also full isolation, as opposed to vPars
          (similar to OpenVZ and appropriate Parallels Virtuozzo solution)





        1. HP
          nPars as part of VSE (cell based virtualization, similar to IBM
          LPARs, HP-UX as hypervisor is only controlling them), demanding
          specific cell-based (NUMA) hardware (rx7640 or stronger, like Superdome)






Pros:
a
highly stable platform which represents on of the leading industry
standards



 
Cons:
the
licenses only for MC OE (DCOE, VSEOE now) are about $10000 per CPU



(TCOE
as a possible minumum is more than $2000 per CPU), performance factor





    1. Xen
      with RHEL5 as hypervisor




 
Good
side: supported
by SAP and HP, Open Source, quite stable and tested, in-house
knowledge available.



            
Cons:
does
not support all the features other options have

                     (at least some of the
features like live migration), performance factor

 

All these solution are stable and robust, but maintenance and support for RHEL / Xen infrastructure is about 1200 euros per server for RHEL Advance Platform (unlimited sockets, unlimited number of guests, cluster suite and GFS included) and that makes it most optimal for system consolidation.

 

*Realization steps*





In general, following implementation gradual
phases and steps are proposed in a short overview:


    1. Prototype
      preparation and testing (in process).
    2. Virtualization
      of all IA64 sandbox (and test) systems and preparation of pool of
      servers (for system copies), Solution Manager (SOD), dialog
      instances of development systems.
    3. In
      parallel, all 32-bit SAP servers on VMWare platform should be
      virtualized (old Solution Manager, IDES, CEN; even SAP routers, but
      not those for end users, for a start).
    4. Complete
      virtualization of development systems.
    5. Test
      drive virtualization for Q&A and production systems (they must
      be aligned, dialog instances first).
    6. Complete
      virtualization of the whole SAP landscape.
Taking Q&A servers together with productive
servers is necessary because of the nature of their intended usage.
The last phase is not yet planned (if it ever will be), at least not
with a final date or specific requirements at the moment (higher
reliability (disaster recovery), greater security and scalability).
This might justify HP-UX as an option as it's implementation has
references nearby, with full implementation and integration support
from HP in a critical environment (with given dates).


For the successful implementation of these phases,
good preparation is crucial, specially about planning needed
resources:




    1. storage
      space




    2. available
      network and optical ports and switches, settings on firewall, etc.




    3. needed
      total number of servers in the landscape (categorized according to
      usage types and associated with possible hosts)




    4. this
      provides information needed to estimate people-days needed for
      implementation of the pool of servers in each of the phases, and
      also physical resources needed for the requested service levels
      according to user requirements.





    1. Much
      of storage space is saved using HP BC Snapshot functionalities (as
      during upgrade), but these also have some level of impact on EVA
      performance and puts constraints on original VLUN, and makes storage
      maintenance more complex. Price of the EVA FC disks is about 15-20
      eur/G which also has to be noted (snapshot grows up to about 10% of
      the original disk size in a normal usage period, which gives around
      1.5-2 eur/G) – there are other storage options, but there also
      other parts of the TCO structure

       


    2. backup
      (about 7 eur/G in averga, but varies from 1 to 15eur)



    3. network
      equipment and security (antivirus, firewall, LAN, FC), licenses



    4. guest
      OS licenses and support agreements



    5. infrastructure
      in the data center room (space, air conditioning, power supply)



    6. system
      maintenance (human resources, licenses for some monitoring
      solutions)



    7. other
      hidden expenses ...





Therefore, it is of the utmost importance to have
business requirements carefully prepared and estimated, because all
real technical requirements (like additional hardware or licenses,
memory, servers, storage). For instance, if large enough number of
servers is hosted (which grows very easily with virtualization), so
grows the I/O bandwidth on SAN or the number of storage spindles.

 

 


*Details
about implementation*




Installation of current environment is divided in three groups:



    1. installation
      of hosts (Dom0s)


    2. installation
      of HVM (fully virtualized) Windows guests


    3. installation
      of PV (paravirtualized) RHEL guests





While the installation of hosts and PV guests has
many similar steps (network setup – I am using trunk utility as
local NTLM proxy for http/ftp, RHN registration, ntfs-3g uitilities
installation, etc), there are things to be set only on hosts (xend
service and boot parameters, HP support pack installation, etc).




For
guests I am generally proposing using copies of the template systems
for many different reasons, one of which is using good features of
EVA storage (snapshots and snapclones) or LVM on local disks. If
making guest from the scratch (or new template), it is always best to
use physical (phy:/) drives instead of files – they have better
performance, enabling moving to another host (with shared storage),
and always use multipathd.





I am giving the following hints for people who
have experience with Windows installations and SAP administration on
Itanium (and who have some basic knowledge and awareness of things
about Linux) – many things are similar, but there are also many
misleading details.

 

 

image

 


 

*Host Installation *



 

    1. RHEL installation is quite simple and straightforward – no EFI boot
      driver is needed (Windows installation needs it), usual server
      preparation, firmware update if needed, boot device configuration in
      EBSU setup disk – boot from SAN is used, so HBA adapater should be
      configured there and a VLUN prepared, after booting from installation
      disk, installer should be started manually by entering “linux
      mpath vnc=1”
      (vnc is optional, GUI is then available later through vnc client on
      screen 1, or http://that-host:5901)
    2. installation number can be entered after the installation (using
      rhn_register), Virtualization group should be chosen if available
      (beside Development, Web Server and others) - or later, packages to
      be installed are: xen, kernel-xen, xen-ia64-guest-firmware,
      virt-manager, libvirt and virt-viewer, or just “yum groupinstall
      Virtualization” (if doing it manually with rpm -ivh in VT folder of
      the installation disk: libvirt-python, python-virtinst,
      Virtualization-en-US, perl-Sys-Virt, xen-devel ../Server/bridge-utils
      ../Server/xen-libs ../Server/gnome-python2-gnomekeyring
      ../Server/iscsi-initiator-utils ../Server/gtk-vnc
      ../Server/gtk-vns-python ../Server/cyrus-sasl-md5)

      ...
      installation lasts up to 1 hour – all setup parameters (like
      network address, gateway, etc) should be prepared

      ...
      check if /boot/efi/efi/redhat/elilo.conf is set correctly before
      restarting

      ...
      /etc/resolv.conf should contain at least one line "nameserver
      DNS_IP_address" and one "domain group.hemofarm.com",
      enabling and disabling network interface is done with ifup eth0 /
      ifdown eth0
    3. on
      some bad terminal emulation/clients after restart, firstboot gets
      stuck (additional setup wizard), which shuts down automatically after
      about 10 minutes, and after logging in it can be disabled with
      “chkconfig firstboot off”
    4. trunk (local NTLM proxy, [http://ntlmaps.sourceforge.net | http://ntlmaps.sourceforge.net/])
      after unpacking with tar -xzf has to be configured
      (NT_DOMAIN:HEMONET, USER:proxyuser, PASSWORD:somepass,
      PARENT_PROXY:proxy_server) and started manually with “scripts/ntlmaps
      &”
      (or put into /etc/rc.local to start automatically after boot), then
      it is possible to start “rhn_register
      –proxy=http://127.0.0.1:5865”
    5. also, for other utilities accessing internet, like yum (package
      installer), edit ~/.bash_profile (using gedit in GUI console, or vi
      editor):

      export
      http_proxy=http://127.0.0.1:5865
      export
      HTTP_PROXY=http://127.0.0.1:5865
      export
      ftp_proxy=http://127.0.0.1:5865
      export
      FTP_PROXY=http://127.0.0.1:5865

      ...
      update can be started with “yum update” after setting up software
      channels and other additional settings on RHN side (if needed)

      NOTE:
      LVM/multipath is not good is not good with FAT partitions, so before
      each kernel update/installation it is needed to mount manually
      /boot/efi (e.g. with “pvs” one can get sytsem disk device for
      system VG LogVol00, like /dev/sd..., for instance /dev/sdh2, and then
      EFI partition is mounted with: “mount -t vfat /dev/sdh1 /boot/efi”)
using Windows (cifs) shares: e.g.
serverx$ can be mounted on /mnt/tmp with:

mount
-t cifs -o dom=HEMONET,username=someuser //server/x$ /mnt/tmp

...
or for simple copying smbclient is useful (like ftp)
    1. ntfs-3g installation (http://www.ntfs-3g.org):

      cp
      /mnt/tmp/rhel/ntfs-3g*.tar
      tar
      xvf ntfs-3g....
      cd
      ntfs-3g...
      ./configure
      make
      make
      install
      yum
      install ntfsprogs
    2. firewall should be configured (opening needed ports,
      system-config-firewall)
      and selinux (for a start and troubleshooting, “setenforce
      0”
      and SELINUX=permissive in /etc/selinux/config)
    3. Xen
      Dom0 configuration for several network adapters, create
      /etc/xen/scritps/network-multi-bridge:

      #!/bin/sh
      dir=$(dirname
      "$0")
      "$dir/network-bridge"
      "$@" vifnum=0 bridge=xenbr0 netdev=eth0
      "$dir/network-bridge"
      "$@" vifnum=1 bridge=xenbr1 netdev=eth1
      "$dir/network-bridge"
      "$@" vifnum=2 bridge=xenbr2 netdev=eth2
      "$dir/network-bridge"
      "$@" vifnum=3 bridge=xenbr3 netdev=eth3

      ...
      and then: “chmod
      +x /etc/xen/scritps/network-multi-bridge”
      and edit /etc/xen/xend-config.sxp, putting “(network-script
      network-multi-bridge)”
      instead of “(network-script
      network-bridge)”

      ...
      also, put there (dom0-min-mem 2048) ... and dom0_mem=2G into
      elilo.conf to reserve 2G for Dom0 and avoid memory ballooning (2G
      should be enough) – elilo.conf example:

      ...
      image=vmlinuz-2.6.18-164.10.1.el5xen
      vmm=xen.gz-2.6.18-164.10.1.el5
      label=2.6.18-164.10.1.el5xen
      initrd=initrd-2.6.18-164.10.1.el5xen.img
      read-only
      root=/dev/VolGroup00/LogVol00
      append="dom0_mem=4224M
      -- palhalt rhgb quiet"
      ...
    4. vncserver setup (/etc/sysconfig/vncservers), vncpasswd should be set
      for the running account, and initial vncserver start (“chkconfig
      vncserver on” for the service, too)
    5. hp
      support pack installation:

      yum
      install net-snmp net-snmp-libs
      ./install.sh
      (options:
      1,2,3,8)
      yum
      install tog-pegasus
      service
      tog-pegasus start

      domainname=group.hemofarm.com
      is needed in /etc/sysconfig/network - in case of additional problems
      with hpsmhd start, in /etc/hosts should be put:

      host-IP-address
      host.group.hemofarm.com host

... initial configuration “/etc/init.d/hpima sample (or better, reconfiguration, especially after each kernel update):

/etc/init.d/hpmgmtbase
reconfigure

...
(and additional restart of services: tog-pegasus, hpmgmtbase, hpima,
hplip, hmsmhd)

/etc/init.d/hpima
reconfigure
/etc/init.d/hpima
start

...
kernel parameters and related settings and recommendations (by
Oracle, SAP):

/etc/rc.local:

ulimit
-u 16384 -n 65536
modprobe
hangcheck-timer hangcheck_tick=30 hangcheck_margin=18

/etc/sysctl.conf:

kernel.msgmni=1024
kernel.sem=1250 256000 100 1024
vm.max_map_count=300000
net.core.rmem_max=1048576
net.core.rmem_default=1048576
net.core.wmem_max=262144
net.core.wmem_default=262144


      1. old
        A/P storages should have excluesively Failover (not failback, not
        None) in the prefered path in the presentation options (mixed environments like I have at the moment are very problematic - upgrade to A/A is more than recommended, it is just about uggrading firmware but needs planning as a potential risk)
      2. multipath.conf example (starting the service and setting chkconfig
        also is needed) - example content of /etc/multipath.conf:






    blacklist {
        devnode "(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"<br />    devnode "(hd|xvd|vd)[a-z]"<br />    wwid ""
    }

    # Make sure our multipath devices are enabled.

    defaults {
        udev_dir /dev
        polling_interval 10
        selector "round-robin 0"
        path_grouping_policy group_by_prio
        getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout "/sbin/mpath_prio_alua /dev/%n"
        path_checker tur
        rr_min_io 100
        rr_weight uniform
        failback immediate
        max_fds 65536
        no_path_retry 12
        user_friendly_name yes
    }

    blacklist_exceptions {
        wwid "3600508b4000129f70002900002030000"
        wwid "36005"<br />    wwid "36001"
    }

    multipaths {
      multipath { /* for newer A/A storages, from EVA VCS4.0 /<br />    wwid 3600508b4000e302d0000a00001200000<br />    alias sap-trn-sys<br />  }<br />  multipath { / for A/P storage, up to EVA VCS4.0 /<br />    wwid 3600508b4000129f70002900000cb0000<br />    path_grouping_policy multibus<br />    path_checker readsector0<br />    prio_callout /bin/true<br />    no_path_retry queue<br />    rr_weight priorities<br />    alias sap-bisem-sys<br />  }<br />  ...<br />}<br /><br />devices {<br />  device {<br />    / EVA 3000/5000 with new firmware, EVA 4000/6000/8000, EVA 4400 /<br />    vendor "(COMPAQ|HP)"<br />    product "HSV1[01][01]|HSV2[01]0|HSV300|HSV450"<br />    getuid_callout "/sbin/scsi_id -g -u -s /block/%n"<br />    prio_callout "/sbin/mpath_prio_alua /dev/%n"<br />    features "0"<br />    hardware_handler "0"<br />    path_selector "round-robin 0"<br />    path_grouping_policy group_by_prio<br />    failback immediate<br />    rr_weight uniform<br />    no_path_retry 12<br />    rr_min_io 100<br />    path_checker tur<br />  }<br />  / MSA */
      device
      {
        vendor "HP"
        product "MSA2[02]12fc|MSA2012i"
        getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
        features "0"
        hardware_handler "0"
        path_selector "round-robin 0"
        path_grouping_policy multibus
        failback immediate
        rr_weight uniform
        no_path_retry 18
        rr_min_io 100
        path_checker tur
      }
    }


      1. Data Protector installation (MUST be started from the PA-RISC disk,
        e.g. B6960-15008-01.tar.gz or downloaded from ITRC SUM):

        yum
        install xinetd
        (if not installed)

        ...
        add port 5555 in iptables firewall (/etc/sysconfig/iptables "-A
        INPUT -m state --state NEW -m tcp -p tcp --dport 5555 -j ACCEPT")

        ./omnisetup
        -source /root/hp/B6960-15008-01/ -server dprotector.group.hemofarm.com -install da,ma,cc

        chkconfig
        --list | grep omni

        ...
        if no service found, edit manually /etc/xinted.d/omni:

        service
        omni
        {
         socket_type
        = stream
         protocol
        = tcp
         wait
        = no
         user
        = root
         server
        = /opt/omni/lbin/inet
         server_args
        = inet -log /var/opt/omni/log/inet.log
         disable
        = no
        }






















    ...
    INFO: linux rescue is a method of disaster recovery, by booting
    installation disk with “linux rescue” or better, “linux rescue
    mpath”, and it is also possilbe to do a reinstallation or upgrade
    over the existing OS file system (as a repair method) by choosing any
    of these options, guided by the installer dialogues



     

    *!!!!! NOTE: always, always make backup of everything that is important ...</p><br />
    *h2. PV Guests

     

     

     

     

    sqlplus / as sysdba ...<br /><br />- SAP:

    su -ertadm
    startsap r3
    stopsap r3

    (using
    all instead of r3 if Java stack is present, as it is with BI systems)


       



        1. shutting down:

          shutdown
          -h 0
          (as with other guest or bare metal systems, there is always a risk
          with forced shutting down of damaging system disk or something more –
          linux rescue boot with prepared sap-rescue configuration with
          affected system disk would help, or restore from a backup ...) or
          init
          0
        2. guests can be backed up just like any other machine using Data Protector or other usual tools (no FC tape medi, though, unless using PCI passthrough which is not available on Itanium even though it has VT-d hardware support)

       

      HVM Guests<br />

       


        1. It
          is enough to copy existing template configuration (e.g..
          /etc/xen/sap-test) for a new guest, having at least changed
          name=new_machine (instead of name=sap-test), and also
          /usr/lib/xen/boot/nvram_sap-test into
          /usr/lib/xen/boot/nvram_new_machine

          (this
          is instead of nvrboot import from the backup on the EFI partition –
          HVM guests have their own EFI environment, just as any bare metal
          machine)
        2. for
          new guest disks created either as snaphots / snapclones or CA
          replicas of the template disks, check the parted /dev/disk/by-id/...
          print before proceeding – if any I/O errors occur, try changing in
          EVA Command View the Presentation / Preferred path/mode on old EVA
          5000 (A or B, when upgraded to new A/A VCS4.xx firmware this will no
          longer be a problem).

          It
          recommended to use for each VLUN it's wwid set accordingly in
          /etc/multipath.conf with appropriate /dev/mapper/alias ...

          With
          ntfsls utility one can even check the M$ partition without any
          mounting
        3. for
          each new disk (which is not a copy/replication, not having gpt label)
          initialization must be done: parted /dev/disk/by-id/... mklabel gpt
          (it is also possible as usual in EFI shell) ...
        4. after Windows boot and administrator login, first the IP address
          should be set in the GUI console (vnc) and remote desktop enabled
          (for a newly installed machine, it should be already set in the
          template) and then continue working with RDP (rdesktop)
        5. the
          next step is changing the name to the machine and adding it to Winows
          domain: newsid /a novoime (restart should be done only from
          virt-manager or with xm create, anything other will fail to bring up
          network correctly)
        6. machine preparation is the same as with any other Windows machine
          (Oracle and SAP installation, mind installing Montecito Oracle
          patches if needed)
        7. e.g. SAP
          ERT 30 instance in the template with preinstalled ERT instance can be
          used only for the same machine, which comes down to same MAC address,
          having same database schema and licenses preserved with (otherwise,
          additional setup is needed):

          ... on
          the old system:
            exp file=lic.dmp userid='/ as sysdba'
          tables=(SAPERP.SAPLIKEY)

          ... on the newly copied system:
           
          imp file=lic.dmp userid='/ as sysdba' fromuser=SAPERP touser=SAPERP





















        1. SAPDATA_HOME and oraarch folders (or whatever set in sapinst for the
          template) are best to be kept on C: - HVM guests have that
          unfortunate constraint to have maximum four drives




      *!!!!!
      NOTE: always, always make backup of everything that is important ...*






       

      *KNOWN
      PROBLEMS*





      I
      had several problems during during all these tests, and these
      are the most significant issues:





        1. EFI
          environment can slow down vnc console and virt-manager refresh
          terribly for unknown reasons (xm commands are usable, but also a bit
          slower, while the guest domains and Dom0 work perfectly normal) –
          a RH service request is open on this one



        2. ORA-07445
          occurs intermittently on Dom0s and DomUs (but does not on normal
          kernel) – a metalink SR is open on this one (bug 8868468) - the
          only viable workarounds are not to use Xen for Oracle, or to somehow
          prepare database without Xen, and then use it under Xen (this is not
          causing serious issues, but it would be unacceptable in producion)







      References




      Note 962334 - Linux: SAP on Xen virtual machine



      Note
      1048303 - Red Hat Enterprise Linux 5.x: Installation and upgrade



      Note
      958253 - SUSE LINUX Enterprise Server 10: Installation notes



      Note
      941735 - SAP memory management for 64-bit Linux systems



      Note
      964705 - Linux IPF: "floating-point assist fault" in Linux
      syslog



      Note
      784391 - SAP support terms and 3rd-party Linux kernel drivers



      Note 527843 - Oracle RAC support in the SAP environment



      Note
      592393 - FAQ: Oracle


       

      http://www.infoworld.com/d/open-source/red-hat-will-drop-itanium-support-in-enterprise-linux-6-083
      http://en.wikipedia.org/wiki/Xen


      Linux Supported platforms
        Supported Platforms
      homogeneous copy x64 to ia64
        Re: System copy  from SuSe Linux-IA64_64 Ent  to SuSe Linux-X8_64 Ent
      SAP on Linux:
        SAP on Linux
        http://www.sdn.sap.com/irj/sdn/index?rid=/webcontent/uuid/e338fb57-0d01-0010-2cb8-e728f66b715f
        http://wiki.sdn.sap.com/wiki/display/HOME/SAPonLinuxNotes
      NovellXenSetup:
        http://wiki.sdn.sap.com/wiki/display/HOME/NovellXenSetup



      http://www.tpc.org/tpcc/results/tpcc_perf_results.asp
      http://www.tpc.org/tpcc/results/tpcc_price_perf_results.asp
      http://ts.fujitsu.com/ps2/press/read/news_details.aspx?id=2202
      http://www.oracle.com/corporate/press/1524171.html
      http://www.nec.co.jp/press/en/0802/2101.html
      http://en.wikipedia.org/wiki/Rate_of_return
      http://en.wikipedia.org/wiki/Total_cost_of_ownership

      SAPS http://www.sap.com/solutions/benchmark/measuring/index.epx


      benchmarks:
      http://www.sap.com/solutions/benchmark/index.epx
      http://service.sap.com/benchmark
      http://www.sap.com/solutions/benchmark/sd3tier.epx?num=200
      http://www.sap.com/solutions/benchmark/sd2tier.epx

      Actions