FTP, securely, securely, from us to SAP

JimSpath · ‎10-23-2015

After a spasm of activity triggered by Jelena 's blog 1 and post 2 [ Are you there, SAP? It's me, Jelena | private ] and my digging through my archives for the last round of "how can we improve SAP software support", I ended up with the task, no, challenge, of evaluating file transfers. Some public (SABOUC conference) and some private (SAP Mentor) discussions revolved around how customers supply data to SAP during support/incident/problem tickets. In the standard ticket queue, you can attach file(s), but you're limited to 4MB--see figure 3 below--small for this day and age. Sending more, larger files, takes an extra step. Files need to be named distinctly, or can't be transferred.

During a support exercise earlier this year, I ran into the size limit, then was given an alternate route, called SAP Mobile Documents (also called SAP Containers). Seemed more like a repository for cell phone apps than a full-fledged file transfer protocol but I gave it a shot. For me, it was a dead end, as the web page I was directed to had nothing but a blank screen. As it turned out later, this was due to browser incompatibility. I'd normally want to see "your browser is not supported" instead of nothing.

If you've got a 4 GB core file sitting on a UNIX box that doesn't have a web browser handy, copying the file to a desktop PC just to turn around and send that to SAP is a hop that I'd want to skip. A GUI file transfer mechanism is only so useful. "Back in the day", we had sapserv4, and we'd send files to some private directory using command line FTP. Simple enough when you use this a lot, and no doubt the GUI incantations make clickers-and-pointers comfortable.

In chats with SAP Support, I learned about "Secure FTP" (also reference the lengthy titled KBA 2077673 below). Turns out this is not the SFTP I use for other transfers, it's known as FTPES, and is not compatible with the ftps software I had on multiple operating systems. The note, or knowledge base article, describes how to use 3 distinct clients for the transfer. One, "FileZilla", is GUI-based, not one that I use routinely, and for many corporate users, maybe not installed or accessible. Like a web browser, perhaps not found on a UNIX system in the data center, nor on a secured Windows system hosting SAP applications. Of the other 2 command line tools, lftp and curl, I had used the latter to set up batch jobs where FTP shell scripting was becoming unmanageable. That was the tool of choice for me.

Curl is venerable, dating back to 1996 (per the source) or 1997 (per Wikipedia), and has more options that you can shake a stick at. And protocols. Lots of protocols. Free software, with no warranty. Perfect. And with vulnerabilities found and exploited seemingly daily, how is one certain they have the right set of tools? On the one hand, you could argue that a core dump file won't contain confidential information of interest to identity or data thieves, but you might not know what data stacks got spooled out. Best to play it safe.

First attempt to send a file, after setting up a practice ticket with SAP's assistance, and building the most recent curl, was a bust.

Versions of curl I found in different places at home and work included 7.19.4, 7.32.0, and 7.42.1 (with libraries up to 7.43.0), and 7.44.0, and protocols from gopher to ldap. Which version will work, which protocol and features must be available would be good to see in the note, as well as security warnings about versions to avoid (ssh, ssl, etc.). I will leave out a tangential discussion of gaining access to a working C compiler on the latest AIX OS. Who owns and deploys these tools would vary from place to place, so I can see this falling between the cracks of the Basis team and OS teams. Normally you would not want most users deploying code that connects to remote systems, though they might need to run it in a hurry when the time comes.

Figuring out why the first curl try failed is next.

$ curl --upload-file TEST.out --ftp-ssl -k --user Jim ftp://x.x.x.x

Enter host password for user 'Jim':

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 ::-- 0:00:05 ::-- 0

curl: (7) Failed connect to x.x.x.x:21; Connection refused

The message "Connection refused" is about as vanilla as it gets. On the bright side, there were no warnings about "unsupported protocol" like I had in my in-house tests:

curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Start, after reading man pages, with command line flags "--verbose" is always good, then the file options considered (--output FILE , --trace FILE , etc.)

-rw-r--r--

1 jim grp

5078 Oct 9 15:08 CURL.TRC

== Info:

SSL certificate verify result: self signed certificate (18), continuing anyway.

Bottom line, the firewall blocks the communication. The SAP KBA/Note says this, in the fine print. Maybe it should be higher up in the preamble ("Have you seen your network security team lately?"). Open an internal ticket, explain the circumstances, supply the SAP note, wait, supply more information, wait again, get the word the firewall has been holed, try again.

While waiting, I tried outside the corporate network. This would be useless in an actual emergency, as you should not be taking critical data out of your network comfort zones. But if the choice was waiting for a ticket to be queued up, or walking over to the nearest wi-fi hot spot, expediencies might dictate. Depends on your risk and comfort level, I suppose. Exhibit A shows a completed upload with the right curl. etc. The speed for this example hardly matters, as I used a 50K test file. If 6 seconds seem long it's probably mostly the protocol handshaking, not the real data being pushed. The network I was on was high speed, these days.

Exhibits B and C are after the firewall red sea parted, for two files of moderate size (3.6 to 6.4 MB). For thoroughness, I should have tested with a 2 - 4 GB core dump, but we tend to delete those off the systems regularly. Oh wait. "That never happens".

Speed

- A -

12-Oct off-premise

$ curl --upload-file file.out --ftp-ssl -k --user i ftp://n.n.n.n

Enter host password for user 'i':

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 49127 0 0 100 49127 0 7850 0:00:06 0:00:06 ::-- 14331

- B -

[...] -20150529.zip --ftp-ssl -k --user i ftp://n.n.n.n

Enter host password for user 'i':

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 3670k 0 0 100 3670k 0 531k 0:00:06 0:00:06 ::-- 990k

- C -

[...] -20150528-20150529.zip --ftp-ssl -k --user i ftp://n.n.n.n

Enter host password for user '':

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 6422k 0 0 100 6422k 0 556k 0:00:11 0:00:11 ::-- 1421k

Back to the Ticket Queue

The challenge to upload files with FTP, securely, is complete. As the context is incident management, what externalities are in play? Others have covered how to open tickets efficiently via other means (see, for example, Selecting the right component below). Searching for FTP may show related hits such as Thomas Jung's blog on making FTP connections from ABAP code, which sends me on a mental tangent on whether the secure modes in play for ticket file transfers has filtered back to the ABAP kernel.

Then there's the wiki page on problem handling that mentions one more file transfer tool, SAP Mail Attachment Service (SAPmats), hinting this is "only SAP Internal" and points to a private URL like sapmats.wdf.sap. "Give me a SAP MATS" is a phrase I might not be able to unlearn.

In the beginning, when you need to open a ticket, you may know the right queue ("here we go again") or it might be new territory ("where do we begin"). Either way, you need to pick a starting place. The workflow shows as [ Choose System ] -> [ Prepare Solution Search ] -> [ Find Solution ]. Easy as pie. When the problem is cross-instance, choosing the system might have you start with the "most affected" rather than the component most likely to be at fault. Then preparing a solution search could be considered duplicate work since you would have searched KBA/Notes, SCN, and probably google too. In my case I had this easy, SAP picked a queue and had someone waiting to pick up the ticket. All problems should have that kind of preparation...

Then there's the priority. Everyone thinks their own issues are top priority. There is production down, and then there's all hands on deck. For my simple exercise, I went with Medium or Low. I've heard some only open High, which tends to diminish the impact ("Crying wolf").

Tracking

Once the ticket is being worked, there are the back-and-forth communications. The advice is upload as much as possible in the beginning, though it's inevitable that either (a) you didn't send something or (b) the technician asks for something already supplied. I'll skip the rant on "ping-pong" and skip to the higher level of "dude, where's my ticket". Two things have not improved since I starting working SAP tickets: (1) the reason a ticket comes back is a mystery, and (2) seeing a management view of ticket aging is not what I'd want. There must be a privacy reason that the email saying SAP has done something merely says:

We want to make you aware of a change to your incident.

The status of your incident was set to "SAP Proposed Solut.".

Please use the following link to access your incident in the SAP Support Portal:

When I get tickets back from other vendors, they include specifics. Sometimes the detail is lacking, but it's generally better than nothing. These days, we see the email on our phones, but depending on how patient you are (and how good your eyesight is), you can view SAP messages on a mobile device. For me, it's a wait for a PC to become available.

Image 2 below, and the snippet immediately following, show the Incident Wizard Action Log. To me, this is a relic of the days of greenbar paper. I hope to see a swim-lane diagram or other visualization showing time spent in states, and a countdown to auto-close.

Closure

Part of what I hoped to test in this exercise was the content of the "exit interview", still known as "Positive Call Closure", where SAP expects customers to give feedback on how the message was handled. This has been addressed via other channels (and I await the latest re-skinning of the support portal to try out the latest flavor), I'll defer. In this instance, I could not close the ticket I had opened. When I started, I cleverly (I thought) decided to use an obscure system reference so that I would not be stepping on toes.

To get the survey, you need to close the ticket. You get an ominous pop-up warning "Closed incidents can no longer be processed" which no doubt has led some people to leave it open "just in case something pops up". If the auto-close timer kicks in you can't do the survey, I've been told. This time, I got the message "You do not have authorization to execute this function", hinting who might repair this. Some back office investigations led to the matrix of green checks, red crosses, and blank spots pictured below. This image was taken after permission was granted; one of those green checks did the trick.

The last image below shows the survey starter panel. Dissection of the topics and process must wait for a followup post.

Summary thoughts

FTP Secure file transfer to SAP has too many wrinkles to be effective, based on my testing and experience
SAP Mobile documents has limited use, though likely the fallback for many
Curl is powerful. Handle with care.
Fill out the survey
The lftp program was new to me, but that should not prevent others from using it
Speed to the SAP target is fine
Did I mention protocols?
Can't wait to see the next generation of file sharing deployed (is it secret; is it safe?)

Apocrypha and edits

While getting this blog fit for daylight, numerous interruptions and delays evinced, none more painful than the Jive software burps. First is the periodic "did I spill your data, oops, my bad":

A recovered version of this content exists from Oct 31, 2015 7:29 PM. Would you like to use the recovered version or delete it?

The second was worse, with the "Are you sure you want to leave this page" trick question that resulted in a royal format hash that lost tables, bullets, and images ending up in rework purgatory.

In the future, will the next Support Platform fix file transfer woes? We shall see.

Links to more and more announcement of the new site look and feel:

Oct-2015 : SAP ONE Support Launchpad: Built on Award-Winning SAP Fiori Technology
Aug-2015 : SAP ONE Support Launchpad Blog Series: A Revolutionary New Way to Access Support
Jul-2015 : SAP Runs Simple with SAP ONE Support Launchpad

Notes and commentary:

Thanks to: danie.theron, mike.trott and kristen.cordell. Plus nameless others.

SCN links

Blogs

SCN Wiki pages

2011 - Problem handling - Problem Handling - Business Server Pages - SCN Wiki
2014 - Select the right component for your incident - Select the right component for your incident - Service Marketplace - SCN Wiki

SAP KBA ("OSS Notes")

Note	Title
93042	Problems with SAPFTP
130106	Using SAPFTP for data transfer
795131	FAQ How to make Secure FTP communication with SAPFTP
797124	LOP - Line Opener Program
895754	Delayed message processing
1896868	How to save a short dump in text format
2077673	SAP Secure FTP server - how to upload files from customer site to SAP Product Support using FileZilla (Windows), lftp (LINUX) or curl (LINUX) as example FTP clients.

HELP with Mobile Documents: SAP Mobile Documents – SAP Help Portal Page

(but more toward running your own repository than how to send a file to SAP, and some info about SAP JAM and SAP Streamwork)

Pictures

0

Ready steady go

1

SAP Container / Mobile Document with unsupported browser:

SAP Mobile Document with newer browser

Document upload progress

Transfer status update

2

3

4

Solution provisioned

But wait