HCI - Using a Twitter use case to understand Apache Camel - Part 1

[bhavesh_kantilal]

Background

I would like to build an Integration Flow on HCI that queries a timeline of a Twitter Handle. I want to make sure Tweets that have been read are not read again. Well this should be easy I thought. There was already a blog series on SCN describing how to set up the Twitter  adapter to post a tweet and a direct message Integrating HCI with Twitter - Part 1 and Integrating HCI with Twitter - Part 2

So, I went about building my Integration Flow as below,

• Start Event: Timer to start the Integration Flow
• Request-Reply: Make the call to the twitter adapter with search keyword set as: from:<twittteracount>
• Response sent back to SFTP Adapter.

Integration Flow

Twitter Adapter

I was skeptical because this configuration did not make sense for multiple reasons,

• What is the response that the Twitter Adapter would return back? Is it XML? What is the format of this data?
• How do I make sure only "Delta Tweets" are provided back to me,i.e, my Integration Flow should only return to me the Tweets I have not read previously. There did not seem to be any option in the Adapter to give me such a handle.

I had nothing to loose and hence triggered my Integration Flow. As expected I did get a error and the error was as below stating: "No type convertor available to convert from type: java.util.ArrayList to the required type java.io.InputStream".

So, twitter did respond back to me but it sent back a Java Array List which meant I had to understand this a little more and then convert this Array List to a XML String. I looked up the documentation from SAP on this but unfortunately there wasn't much to go on here. So, how do we proceed from here? What could be the content of this ArrayList I wondered.

HCI uses the Apache Camel  framework is something that all of us are aware of. When trying to figure out the answer to this question on Twitter adapter I decided to check Apache Camel on this and see if there was a means to reverse engineer the workings of the HCI twitter adapter

Setting up the Built in Examples of Apache Camel

Apache Camel: Examples provides with a comprehensive list of examples that can be used to trigger an Apache Camel flow and understand its underlying components. One of the examples in this page listed was the Twitter Websocket Example.

The example is demonstrating how to poll a constant feed of twitter searches and publish results in real time using web socket to a web page

Exactly what I want was my first reaction. So how do I run this example locally? How do i test and play around with this example to understand the the HCI twitter adapter.

In part 1 of this blog series, we will set up this example locally and test the same.

In part 2, we will extend the learning from this standalone run to HCI and use the same in HCI Twitter Adapter.

High Level Steps

1. Download and Install Apache Camel -  Apache Camel: Download
2. Download and Install Apache Maven - Download Apache Maven
3. Set Up your Environment Variables
4. Download project using Maven
5. Import project into Eclipse
6. Run & Understand the Project

1. Download and Install Apache Camel

Step	Details
Download Apache Camel Distribution from link Apache Camel: Download In this case we will download the Windows Distribution for Camel 2.17.x
Unzip the Contents of the zip file to any directory. In this case I extract the same to my E:\ Note: Make sure your directory names do not have any spaces in them if you plan to extract this into any other location. There are some known issues when you try to start Camel in a directory with spaces.

2. Download and Install Apache Maven

Step	Details
Download Apache Maven from the link - Download Apache Maven
Unzip the Contents of the zip file to any directory. In this case I extracted the same to my E:

3.Set Up your Environment Variables

Step	Details
Set up JAVA_HOME by pointing to your JRE Directory. Right Click on Computer --> Properties --> Advanced System Settings -> Environment Variables JAVA_HOME = C:\Program Files\Java\jre7 ( In my case ) Make sure you have Java Version >= 7.0
Set up your Path Variable to point to the Maven Bin Directory Maven Bin Directory = E:\apache-maven-3.3.9-bin\apache-maven-3.3.9\bin
Verify Maven is Ok by launching Command Prompt and executing command - mvn -v
Verify JAVA_HOME by launching command prompt and executing command echo %JAVA_HOME%

4. Download Project Using Maven

Step	Details
Navigate to your Camel Directory --> examples --> camel-example-twitter-websocket In my case : E:\apache-camel-2.17.0\examples\camel-example-twitter-websocket Note the path down as this will be used in your Command Line Prompt
Launch Command Prompt Navigate to the Camel Directory
Execute Command - mvn eclipse:eclipse
Your Project will now be downloaded. You may get some warnings - they can be ignored You should have the message "BUILD SUCCESS"
Navigate to your directory : E:\apache-camel-2.17.0\examples\camel-example-twitter-websocket You should now see the Java Project with the source code, classpath etc.

5. Import Project into Eclipse

Step	Details
Open Eclipse, right click on Package Explorer, import --> Existing Projects into Workspace
Select the directory where the maven project was downloaded. In this case, E:\apache-camel-2.17.0\examples\camel-example-twitter-websocket Click Finish
Your project should be listed as below

6. Run  and Understand the Project

Step	Details
Navigate to CamelTwitterWebSocketMain.java under src/main/java The Java Class uses CamelTwitter Account as its twitter account This Example publishes the tweets with the word "gaga" into a HTTP Socket running on URL : http://localhost:9090/index.html . As we are interested in the workings of the Apache Camel Twitter Component we will not focus on viewing the tweets in the browser. Run this as-is without any changes ( Run -> RunAs-> Java Application ) You should be able to view the resulting tweets in the Console
Twitter limits the number of calls in its API and hence after a point this will start having errors as shown. Stop your Java Program
Currently the code searches for the term : "Gaga" For the  current requirement to read the tweet handle of a particular user, this can be changed to "from:<username>"
Update your Twitter credentials if required

So Far So Good But..

So far so good, I thought! From the looks of it, it was clear that HCI was using the Twitter Component of Apache Camel as its underlying framework for the Twitter adapter. But, I still didn't have an answer as to how to restrict the twitter feed to only new tweets. I still do not understand what the Java Array List is all about.

I went about then digging on Apache Camel Twitter component documentation - Apache Camel: Twitter. Couple things were of interest here,

Apache Camel Documentation Excerpt	Inference
The Twitter component enables the most useful features of the Twitter API by encapsulating Twitter4J.	The Twitter component of Apache Camel in turn using Twitter4J Would recommend reading through Twitter4J and what it does on the link provided at a high level.
sinceId - Camel 2.11.0: The last tweet id which will be used for pulling the tweets. It is useful when the camel route is restarted after a long running.	sinceId was the answer to how to keep track of the tweets that have already been read..What does it actually do? Another round of reading led me to twitter's official documentation on this :https://dev.twitter.com/rest/public/timelines. Would recommend reading this link to understand the working of sinceId.
	The search function returned a List of Type <twitter4j.Status> . This meant I could now understand what the search function returns and read data as I need by looking up the Java Doc of Twitter4j.status Status

So,in summary, it was clear to me now that I had to use a combination of Twitter4j.status and sinceID to build what was needed to meet my requirement.

I had been able to reach this point by running a built in apache camel twitter example and understanding the underlying working of HCI's twitter adapter. In the next part of this blog, we will look at how the info deciphered above is used in our Integration Flow.