At the 2012 Aussie Demo Jam, held as part of the SAP Inside Track Sydney and Mastering SAP Technologies event on March 25th 2012, I showcased a demo that used a Parrot AR.Drone helicopter, ABAP and the Siri voice activated assistant that comes on the latest Apple iPhone 4S.
Since then many people have asked varying questions about what was shown, how it worked, how it was built and why you would do it anyway.
This associated blog is a bit of a ramble about how I came to do this demo - and why it wasn't part of the Demo Jam competition itself.
And in this very long post I will try as best as I can to document how the demo was built. After each section I have a YouTube video to show that piece of the demo. If you are short of patience you can view the full Siri.Drone demo at http://www.youtube.com/watch?v=Ek1J7gq1nF8.
The whole premise of the demo was that I wanted to show how I could use the SAP ABAP stack as a mediator between contemporary consumer style clients, in this case the Siri voice assistant that comes with the Apple iPhone 4S, and other non-SAP applications such as the Parrot AR.Drone.
*Update: Parrot have stopped selling the AR Drone so their information is not being maintained. This link takes you to a review of Parrot's AR2 drones thanks to Chris Brooks.
Step 1: ABAP Interface to AR.Drone
So first and foremost we have to be able to interact with the AR.Drone from the SAP WAS ABAP. We need to be able to send commands to the drone and receive the navigation data from the drone.
While technically it might be possible to build an SDK in ABAP that would establish and manage the communication path to and from the drone, provide a command interface to the drone and a facility for receiving data from the drone this would be a lot of work.
Instead I remembered a recent customer engagement of mine where, rather than build some complex algorithm in ABAP, I used the JCo connector to call an existing Java program that had already implemented the required algorithm.
Whilst to date Parrot have only provided SDKs for iOS and Android I found a Windows SDK for the AR.Drone built by Stephen Hobley, Thomas Endres and Julian Vinel. These guys have used this SDK to also build a Windows control application for the AR.Drone and they have released the source code on GitHub. This AR.Drone control program uses the keyboard and DirectX inputs like joysticks and the Wiimote to send commands to the drone. It also supports the display and saving of the video feed from the drone onboard cameras.
This short video demonstrates the control program.
It quickly became apparent to me that the AR.Drone Control program already did all the heavy lifting in terms of establishing the connection with the drone, sending it commands, receiving navdata and video data. It therefore made sense for me to simply establish a connection to this program rather than reinvent the wheel by trying to build a complete SDK for ABAP. (Thanks Stephen, Thomas & Julian - you saved me heaps of trouble.)
This is quite easy to do using the SAP .Net Connector. You can refer to this great blog from Thomas Weiss that pretty much shows you everything you need to do to setup a .Net program as a RFC client or server. In my case I needed the .Net program to be the RFC Server that could be consumed by an ABAP RFC client. I used Thomas' blog as the main reference as I modified the AR.Drone Control Program to make it an RFC Server.
Rather than have multiple interfaces for each operation I added a single RFC handler to the AR.Drone Control Program that could take input commands and pass them to the existing keyboard driver and optionally return the current navigation data to the calling program.
As Thomas explains in his blog the SAP .Net Connector can dynamically lookup metadata in the ABAP system at runtime. This removes the need for regenerating .Net proxies, or in fact for explicitly creating them at all. This may seem counter-intuitive when we are building a RFC server program rather, than a client program, but it becomes very useful for fault finding if you try and use a return signature that does not conform to the signature of the ABAP function module. The runtime error codes and messages are quite explicit about what you have done wrong.
I created a proxy RFC-enabled function module in my ABAP system called ZARDRONE_PROXY. This function module contains no code, as we never run it, the important thing is the function module signature that is introspected by the AR.Drone Control program at runtime.
FUNCTION ZARDRONE_PROXY.*"----------------------------------------------------------------------*"*"Local Interface:*" IMPORTING*" VALUE(COMMAND) TYPE STRING*" EXPORTING*" VALUE(ALTITUDE) TYPE ZARDRONE_NAVDATA-ALTITUDE*" VALUE(BATTERYLEVEL) TYPE ZARDRONE_NAVDATA-BATTERYLEVEL*" VALUE(PHI) TYPE ZARDRONE_NAVDATA-PHI*" VALUE(PSI) TYPE ZARDRONE_NAVDATA-PSI*" VALUE(THETA) TYPE ZARDRONE_NAVDATA-THETA*" VALUE(VX) TYPE ZARDRONE_NAVDATA-VX*" VALUE(VY) TYPE ZARDRONE_NAVDATA-VY*" VALUE(VZ) TYPE ZARDRONE_NAVDATA-VZ*" VALUE(IMAGEFILE) TYPE ZARDRONE_NAVDATA-IMAGEFILE*"----------------------------------------------------------------------" This is just a proxy function module to provide the interface for the
" .Net Connector at runtime.ENDFUNCTION.I have dumbed down the interface as much as possible to avoid playing around too much with datatype conversion - in a real production use case you would pay much more attention to this area. The ZARDRONE_NAVDATA structure is defined like this.
| Component | Data Type | Length | Decimals | Short Description |
|---|---|---|---|---|
| ALTITUDE | INT4 | 10 | 0 | Integer type from .Net Connector |
| BATTERYLEVEL | INT4 | 10 | 0 | Integer type from .Net Connector |
| PHI | FLTP | 16 | 16 | Double type from .Net Connector |
| PSI | FLTP | 16 | 16 | Double type from .Net Connector |
| THETA | FLTP | 16 | 16 | Double type from .Net Connector |
| VX | FLTP | 16 | 16 | Double type from .Net Connector |
| VY | FLTP | 16 | 16 | Double type from .Net Connector |
| VZ | FLTP | 16 | 16 | Double type from .Net Connector |
| IMAGEFILE | STRING | 0 | 0 | String |
This is essentially a replication of the standard data structure for navigation data provided by the AR.Drone SDK. I have added a field for IMAGEFILE to pass back a snapshot image filename when requested.
This is what my .Net code looks like in the RFC handler.
public class RFC_Handler
{
[RfcServerFunction(Name = "ZARDRONE_PROXY")]
public static void ArdroidProxy(RfcServerContext context, IRfcFunction function) {
String command = function.GetString("COMMAND");
RfcInput.addCommand(command);
if (droneControl != null) {
function.SetValue("ALTITUDE", droneControl.NavigationData.altitude);
function.SetValue("BATTERYLEVEL",droneControl.NavigationData.batteryLevel);
function.SetValue("PHI", droneControl.NavigationData.phi);
function.SetValue("PSI", droneControl.NavigationData.psi);
function.SetValue("THETA", droneControl.NavigationData.theta);
function.SetValue("VX", droneControl.NavigationData.vX);
function.SetValue("VY", droneControl.NavigationData.vY);
function.SetValue("VZ", droneControl.NavigationData.vZ);
function.SetValue("IMAGEFILE", "snapshot1.png" );
}
if (!context.Stateful) context.SetStateful(true);
}
}I modified the standard class provided in the program for keyboard input for my purpose and called it RfcInput. The handler simply gets the value of the input parameter COMMAND and passes it as if the user had pressed it on the keyboard. It then reads the latest navigation data from the droneControl object and returns it to the caller.
So now when I start the AR.Drone Control program it registers itself with my ABAP system as an RFC Server called ARDRONE. ( The AR.Drone Control program talks to the drone via Wifi in ad-hoc mode. So my laptop uses the ethernet connection to connect to the network, and therefore my SAP system, and the Wifi connection to connect to the drone. The AR.Drone Control .Net program runs on this laptop. )
This means I can now call the handler just as I would any other RFC. I created a class called ZCL_ARDRONE_CONTROL to encapsulate all the code that interacts with the .Net program. I have a static private method called SEND_COMMAND that actually implements the RFC call to the AR.Drone Control program.
METHOD send_command.
CALL FUNCTION 'ZARDRONE_PROXY'
DESTINATION 'ARDRONE'
EXPORTING
command = command
IMPORTING
altitude = navdata-altitude
batterylevel = navdata-batterylevel
phi = navdata-phi
psi = navdata-psi
theta = navdata-theta
vx = navdata-vx
vy = navdata-vy
vz = navdata-vz
EXCEPTIONS
system_failure = 1
communication_failure = 2
OTHERS = 3.
ENDMETHOD.This is just a demo so I haven't spent any time implementing error handling. NAVDATA is a readonly static public attribute so it always contains the latest values and is readable from outside the class.
I have defined all the keyboard commands as private constants. So, for example, CO_LAUNCH is set to 'Return' and CO_HOVER is set to 'NumPad0' which are the default keyboard keys that send the LAUNCH and HOVER commands.
Then for each command I have a static public method that calls the SEND_COMMAND method passing the appropriate command.
METHOD launch.
send_command( co_launch ).
ENDMETHOD.
So now I can send each command I want by testing each method.
Step 2: ABAP UI for AR.Drone
Now that we have successfully established the connection between the ABAP system and the drone via the AR.Drone Control program the next step is to build a user interface on the ABAP stack.
I needed something that looked great, was lightweight, performed well and that I could easily adjust as I came up with new ideas about what to put into it. No one who knows me would be surprised that I chose to do this as a Business Server Page application.
I think that the Internet Communication Framework is one of the most valuable pieces of the SAP ABAP application server. When SAP thought up the ICF it was a real "ah-ha" moment. To make building ICF services easy SAP also delivered the BSP framework to provide a server-side scripting programming model for building web applications.
I liked the look of the UI that the AR.Drone control program provided and decided to try and do something similar.
When I looked into the code for the AR.Drone Control program to see how they did the cockpit instrumentation I found that all the gauges were image files that were just repositioned as required to reflect the current navigation data from the drone. All the images that were used, and some extra ones as well, were delivered with the program code so I decided to use the same images to make my helicopter cockpit have the same sort of look as the AR.Drone Control program.
Using the compass as an example, I found that this gauge was made up of three different images that would be placed on top of each other.
 |
| Heading_Background.gif |
 |
| Heading_Wheel.gif |
 |
| Heading_Aircraft.gif |
Let me show you an example of how this is done using HTML, Javascript and CSS.
The first job is to position the images correctly. So if I declare the images using this HTML…
<div id="compass">
<span id="compassbackground"><img src="images/Heading_Background.gif"></span>
<span id="compasswheel"><img src="images/Heading_Weel.gif"></span>
<span id="compassneedle"><img src="images/Heading_Aircraft.gif"></span>
</div>
…I can use stylesheet entries like this to position the images on top of each other and prepare the compass wheel for rotation.
<style>
#compasswheel
{
position:relative;
left:13px;
top:-291px;
z-index:15;
-webkit-transform: rotateZ(360deg); /* Pre-rotate 360 degrees */
-webkit-transition: -webkit-transform 1s linear; /* Smooth 1 second transitions */
-webkit-transform-origin: 14.5% 50%; /* Set origin of transformation */
}
#compassneedle
{
position:relative;
left:70px;
top:-545px;
z-index:20;
}
</style>
You will need to use appropriate settings for your specific document if you try this yourself.
* Hint - the developer tools that are built into Google Chrome make figuring out the correct stylesheet settings to position the images a breeze.
Note that I am using Webkit CSS properties for the rotation. I have only tested this example with Google Chrome so I'm sorry if it doesn't work for you. You will just need to figure out the appropriate stylesheet entries for your browser.
The z-index CSS property specifies the stack order of each element. An element with a greater stack order is always displayed in front of an element with a lower stack order.
Once the images are properly located it is simply a matter of rotating the HeadingWheel.gif image the appropriate amount to reflect the current drone heading. I created the following JavaScript function to do this by simply passing it the required compass heading.
function set_compass(deg)
{
$('#compasswheel').css('webkitTransform','rotateZ('+String(360-deg)+'deg)');
}
I found that by pre-rotating the compass wheel 360 degrees and then applying this offset each time I perform further rotations I achieved a smooth transition when the heading moved between positive and negative values.
Although some of the gauges are more complicated than the compass example it did not take very long for me to build a complete set of instrumentation that I deployed in a BSP page I called avionics.htm.
The steps I took were first to created a BSP application called ZARDRONE and an associated application class called ZCL_ARDRONE_BSP_APP. I defined this as a stateful BSP application so things like the ZCL_ARDRONE_CONTROL class and the RFC connection to the AR.Drone Control program would instantiate once and then be reused.
Then I created and built my avionics page. I have added a few more gauges just because I could.
The next step was to link the cockpit instrumentation in the avionics.htm page to the latest navigation data retrieved from the drone.
To do this I created a simple BSP page called command.json. The purpose of this page was to act as the entry point for dispatching commands via my ZCL_ARDRONE_CONTROL class and to return the latest navigation data. I defined a dummy command called REFRESH that the AR.Drone control program would simply discard.
In the IF_BSP_APPLICATION_EVENTS~ON_REQUEST method of my application class I intercepted any requests directed to command.json.
METHOD if_bsp_application_events~on_request.CASE runtime->page_name.
WHEN 'command.json'.
send_command( request ).
WHEN OTHERS.
ENDCASE.
ENDMETHOD.
The SEND_COMMAND method attempts to call the method of the same name as the COMMAND parameter in the ZCL_ARDRONE_CONTROL class.
METHOD send_command.
DATA: command TYPE char20.command = request->get_form_field( 'command' ).
CHECK command IS NOT INITIAL.
TRANSLATE command to UPPER CASE.
TRY.
CALL METHOD zcl_ardrone_control=>(command).
CATCH cx_root.
ENDTRY.
ENDMETHOD.
In the layout of the command.json page I put the following code. This fills the response payload with the latest navigation data in JSON format.
<%@page language="abap"%>
{
"navdata": {
"altitude": <%= zcl_ardrone_control=>navdata-altitude %>,
"batterylevel": <%= zcl_ardrone_control=>navdata-batterylevel %>,
"phi": <%= zcl_ardrone_control=>navdata-phi %>,
"psi": <%= zcl_ardrone_control=>navdata-psi %>,
"theta": <%= zcl_ardrone_control=>navdata-theta %>,
"vx": <%= zcl_ardrone_control=>navdata-vx %>,
"vy": <%= zcl_ardrone_control=>navdata-vy %>,
"vz": <%= zcl_ardrone_control=>navdata-vz %>,
"snapshot": <%= zcl_ardrone_control=>navdata-imagefile %>
}
}In the Avionics BSP page I created this javascript function, which is called immediately after the page is loaded, to make an asynchronous HTTP call to the command.json page every 100 milliseconds thereby constantly updating the avionics with the latest navigation data. This function makes use of the Ajax and JSON parsing capabilities of the JQuery JavaScript Library.
function update_navdata()
{
$.getJSON('command.json?command=REFRESH', function(data){
set_compass(data.navdata.psi);
set_altimeter(data.navdata.altitude);
set_pitch_angle(data.navdata.theta);
set_roll_angle(data.navdata.phi);
$(this).delay(100);
update_navdata();
});
} Note how the compass heading is stored in the PSI value. This value is passed to our set_compass function. In the same way all the other navigation data values are passed to functions that in turn change the appropriate cockpit gauge settings.
Last step was to add some buttons to the UI for sending commands to the drone, a suitably dressed pilot, and the AR.Drone BSP Control screen was done.
You will notice when you watch the video that I have also started to add some of my own commands, like UP and DOWN. These commands adjust the height by a predetermined amount. So rather than just send a command to the drone we send a GAZ (acceleration) command and then monitor the altitude until it has reached the required value when we put the drone into hover mode. These "macro" commands are all implemented in the AR.Drone Control program itself.
Step 3: Service Enable AR.Drone ABAP Control Program
I decided just to use a simple REST style interface to service enable the ABAP AR.Drone control program.
I used D.J. Adams Alternative Dispatcher Layer as the basis for an ICF handler to implement the service layer. You can learn about and download the ADL from SAP Code Exchange.
The ICF dispatcher and subsequent handler are built using the same design pattern as DJ subscribes to in the ADL. My handler class is passed the command to be sent to be sent to the AR.Drone Control program. It then just calls the appropriate method of the ZCL_ARDRONE_CONTROL class, in the same way the BSP application class does, and returns a simple success message.
TRY.
CALL METHOD zcl_ardrone_control=>(command).
CATCH cx_root.
ENDTRY.
What could be simpler? ADL Rocks!
Step 4: Siri Proxy
The SiriProxy project is a contribution from Pete Lamonica and published under the GNU GPL license version 3. You can find all the details here on GitHub. This includes all the source, extensive documentation and even a video showing how to install it.
I installed my SiriProxy on a Ubuntu Linux server running as a Parallels virtual machine on my Mac.
The setup is really very straight forward. You just need to intercept the DNS requests from the iPhone 4S for the Apple server that normally handles Siri requests and respond with the IP address of the hostname of the host running the SiriProxy server.
Pete's instructions cover all the details including how to generate the required digital certificate and install it on the iPhone 4S.
SiriProxy runs on Ruby and you create your own plug-ins for handling Siri commands - again Pete's instructions cover this in detail with lots of examples. And as for Ruby syntax you simply have to Google it.
I created my own plug-in for the drone commands I wanted to implement and just let all other commands pass through to Apple for normal processing. Where I detected a command I wanted to process I simply called the SAP ICF service handler with that command and then passed the response back to the iPhone.
This is an example of the what the service call might look like in my SiriProxy plug-in.
def launch
Thread.new {
page = HTTParty.get("http://#{self.endpoint}/launch).body rescue nil
status = JSON.parse(page) rescue nilif status
say "#{status.["response"]}."
else
say "Communication Error"
end
request_completed
}
end
You can see how I make an HTTP call to the web service passing the command (in this example "launch") and then parse the JSON response payload to extract it's contents and pass it back to Siri to display it and read it out loud.
- SAP Managed Tags:
24 Comments
| User | Count |
|---|---|
| 6 | |
| 5 | |
| 3 | |
| 3 | |
| 2 | |
| 2 | |
| 2 | |
| 1 | |
| 1 | |
| 1 |
