on 10-26-2010 1:53 PM
Hi All,
I am facing some problem with encoding issue.
Scenario : JMS --> SAP PI --> JMS
Requirment : Input plain text file contain some special characters,"©®" . Based on this condition,In Java Mapping
we check the Payload and changed the 'encoding' tag to UTF-8 or ISO8859-1.
: <?xml version="1.0" encoding="UTF-8"?> in the target XML output.
While testing in Operation mapping our Java mapping works fine. as the encodeing tag changes from
UTF-8 to ISO8859-1 if the special character exists.But if I test the same in Integration Directory(Test Configuration)
or did a end to end testing. The encoding tag did'nt changes.
For testing we had to a set of Plain Text files with UTF-8 and ISO8859-1 .
I tried the options of using beans in Adapter modules in Sender JMS channel.
MessageTransformBean, TextCodepageConversionBean, XmlAnonymizerBean
These doc & threads ,was also referred[How to Handle Encoding in PI|http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/502991a2-45d9-2910-d99f-8aba5d79fb42]
Regards,
Ashutosh R
Did you try with XSLT mapping ?
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" encoding="ISO-8859-1"/>
<xsl:template match="/">
<xsl:copy-of select="*" />
</xsl:template>
</xsl:stylesheet>
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Why you change the encoding flage based on a value?
I assume that the encoding is always iso8859-1.
The standard beans do not work as they assume the encoding in the content-type of the message (something like "text/plain;charset=iso8859-1")
But as this is an HTTP header parameter, it does not exist in JMS sender adapter.
As workaround you can use the MTB twice.
For the first you set Transfer.ContentType to text/plain;charset=iso8859-1
For the second (where you use also FCC) you set Transfer.ContentType to text/plain;charset=UTF-8
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Stefan.
Sorry for the late reply....
Why you change the encoding flage based on a value?
we need to send target XML file with appropriate tag value based upon
special character " © ®..
if © ®. is present in data, encoding should be iso-8859-1 else UTF-8.
The issue i am facing is : The required functionality work by Java mapping and tested out of PI box
Tested same result in Operation mapping Test ..
But while testing end to end or Integration directory ( Test configuration)
the result varies... i mean for data not having " © ®." it create the encoding tag value as
Iso-8859-1. which should be utf-8...
Is in one of pipeline steps after operation mapping , the Source file is Flat text file, with no encoding details, Only PI " © ®" if else option will decide the target xml file encoding...apart from other mapping transformations.
Regards,
Ashutosh
and what add to my confusion is the SXMB_MONI trace steps snapshots
<Trace level="2" type="T"> Call method execute of the application Java mapping ESADeploy.ModifyOrg123456 </Trace>
<Trace level="1" type="T">*** START APPLICATION TRACE ***</Trace>
<Trace level="2" type="T">Process Started</Trace>
<Trace level="1" type="T">*** END APPLICATION TRACE ***</Trace>
<Trace level="3" type="T">Search esawarehouse/input_propertyfile.xml (http://xx.com/Logistics/ESAWarehouse, -1) in swcv 1c74ad40-cf77-11de-9c50-d0f1346358c2.</Trace>
<Trace level="3" type="T">Loaded resource esawarehouse/input_propertyfile.xml</Trace>
<Trace level="2" type="T"> Java mapping ESADeploy/ModifyOrg123456 completed. (executeStep() of ESADeploy.ModifyOrg123456). </Trace>
<Trace level="3" type="T">Dynamic Configuration ( http://sap.com/xi/XI/Message/30/general senderAgreementGUID b2465081fad9321----- )</Trace>
<Trace level="1" type="T"> ContentType text/plain;charset=UTF-8 </Trace>
</Trace>
</Trace>
The last step is no where Sync with the result of the Java mapping output of <encoding> tag value...
no idea why this step executes...
Regards,
Ashu
> But I am using Java Mapping, and testing it in Operation mapping level.
Maybe I did not understand the issue correctly.
After your Java mapping, the encoding tag of the XML header should be the same as you have coded.
Is this not the case? Besides mapping the payload is not changed inside XI runtime.
Hi stefan,
Yes, the Java Mapping changes the <encoding> tag value as coded.only in , while testing in operation mapping tab.
BUT when i execute the scenario end to end with same set of test data, the <encoding> tag value is not , what it is expected.
even i tried to test the interface in integration directory ( Test configuration).. the <encoding> tag value is unexpected....
Regards,
Ashu
I think in test tab you use a different codepage for the characters compared to the original files.
You have to consider in your Java program that the "©®" characters come in ISO codepage, so you cannot simple do a string compare.
You have to copy the inputstream to a byte array and check for byte values of the iso codepage.
Maybe you post your java program, so we can discuss it.
Your concern is viable but One thing to recall.
Result is ok in Operation mapping test tab
Result unexpected :
1. Integration directory ( Test configuration ) : Test data is same as for Operation map testing...
2. End to end testing.
i.e why, my concern was with any pipeline steps since Integration directory ( Test configuration ) test the whole interface..
will paste the code... and flow details...
regards,
Ashu
Hi,
The code excceds the max limit of 150000chars. i can mail it to u..
still the fraction snapshot is :
//******************Start of Code Parse the input XML******************//
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
Document documentIn = null;
DocumentBuilder builder = null;
try {
builder = factory.newDocumentBuilder();
} catch (ParserConfigurationException e1) {
e1.printStackTrace();
}
try {
documentIn = builder.parse(in);
} catch (SAXException ex) {
Logger.getLogger(ModifyOrg1234567.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(ModifyOrg1234567.class.getName()).log(Level.SEVERE, null, ex);
}
NodeList list_f01=documentIn.getElementsByTagName("body");
String str_f="";
for(int k=0; k<list_f01.getLength();k++){
Node f=list_f01.item(k);
f=f.getFirstChild();
str_f= str_f+f.getNodeValue();
}
//******************End of Code Parse the input XML******************//
*************Start propery file lookup***********
FileInputStream inPropFile1 = new FileInputStream("C:/Documents and Settings/rawata1/Desktop/ESA WareHouse/input_propertyfile.xml");//
//InputStream inPropFile1 = this.getClass().getClassLoader().getResourceAsStream("esawarehouse/input_propertyfile.xml");
Properties properties = new Properties();
try {
properties.loadFromXML(inPropFile1);
keytagvalueTokenOne = properties.getProperty(this.tokenOne);
keytagvalueTokenTwo = properties.getProperty(this.tokenTwo);
keytagvalueTokenThree = properties.getProperty(this.tokenThree);
********************end look up**************************
******Call search method********
encodingType = fixSpecialCharforWeb(fileContent);
**********Setting the encoding tag value :**************
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transform = tf.newTransformer();
if (encodingType == true) {
transform.setOutputProperty(OutputKeys.ENCODING, "iso-8859-1");
} else if (encodingType == false)
{
transform.setOutputProperty(OutputKeys.ENCODING, "utf-8");
}
System.out.println("ashutosh transformer executed");
transform.transform(new DOMSource(documentOut),new StreamResult(out));
//out.write(inptxml.getBytes());
Read the input xml file, the whole data is stored in a String "fileContent".
The "fileContent" data is also passed to method for Special Character Searching
fixSpecialCharforWeb();which decides the encoding tag value.
Their is a property File lookup is done to get some token values
then an whole series of String concatenation creates a target xml.
The
transform.setOutputPropert
y is used, based upon the Search method output.
I even use even o
ut.write(inptxml.getBytes(utf-8 or iso8859)
inplace of
setOutputProperty.program works but again end to end testing and ID(test config ) give unexpected
results.
So i am back to zero...
Regards,
Ashu
Hi
public static boolean fixSpecialCharforWeb(String text) {
int i = 0;
Character c = null;
char[] ctext = null;
StringBuffer newText = new StringBuffer("");
//boolean encodingType = false;
if ((text == null) || (text.trim().length() == 0)) {
return encodingType;
} else {
try {
for (i = 0; i < text.trim().length(); i++) {
ctext = text.trim().substring(i, i + 1).toCharArray();
c = new Character(ctext[0]);
//Single quote
if ((text.trim().substring(i, i + 1).equals("'")) || (c.hashCode() == 8217) || (text.trim().substring(i, i + 1).equals("?")) || (c.hashCode() == 146) || (c.hashCode() == 145)) {
//newText.append("'");
encodingType = true;
return encodingType;
}
//Double quotes
if ((c.hashCode() == 8220) || (c.hashCode() == 8221) || (c.hashCode() == 147) || (c.hashCode() == 148)) {
//newText.append(""");
encodingType = true;
return encodingType;
}
// bullet point
if ((c.hashCode() == 8226) || (c.hashCode() == 149)){
encodingType = true;
return encodingType;
}
// tilde
if ((c.hashCode() == 732) || (c.hashCode() == 152)){
encodingType = true;
return encodingType;
}
// Soft Hypen
if (c.hashCode() == 173){
encodingType = true;
return encodingType;
}
// En-Dash
if ((c.hashCode() == 8211) || (c.hashCode() == 150)) {
encodingType = true;
return encodingType;
}
// Em-Dash
if ((c.hashCode() == 8212) || (c.hashCode() == 151)) {
encodingType = true;
return encodingType;
}
// Euro Sign
if ((c.hashCode() == 8364) || (c.hashCode() == 128)) {
encodingType = true;
return encodingType;
}
// Yen Sign
if (c.hashCode() == 165) {
encodingType = true;
return encodingType;
}
// Pound Sign
if (c.hashCode() == 163) {
encodingType = true;
return encodingType;
}
// 1/2 sign
if (c.hashCode() == 189) {
encodingType = true;
return encodingType;
}
// 1/4 sign
if (c.hashCode() == 188) {
encodingType = true;
return encodingType;
}
// 3/4 sign
if (c.hashCode() == 190) {
encodingType = true;
return encodingType;
}
// Sword/dagger
if ((c.hashCode() == 8224) || (c.hashCode() == 134)) {
encodingType = true;
return encodingType;
}
// Trademark
if ((c.hashCode() == 8482) || (c.hashCode() == 153)) {
encodingType = true;
return encodingType;
}
// Ampersand &
if ((text.trim().substring(i, i+1).equals("&")) || (c.hashCode() == 38)) {
encodingType = true;
return encodingType;
}
//Registered mark
if ((text.trim().substring(i, i + 1).equals("?")) || (c.hashCode() == 174)) {
//newText.append("®");
encodingType = true;
return encodingType;
}
//Copyright mark
if ((text.trim().substring(i, i + 1).equals("?")) || (c.hashCode() == 169)) {
encodingType = true;
return encodingType;
}
// Question.
if (c.hashCode() == 63 && c.toString().equals("?")){
//newText.append("?");
encodingType = true;
return encodingType;
}
//handling symbol ?
if ((text.trim().substring(i, i+1).equals("?")) || (c.hashCode() == 233)) {
encodingType = true;
return encodingType;
}
//?
if ((text.trim().substring(i, i+1).equals("?")) || (c.hashCode() == 232)) {
encodingType = true;
return encodingType;
}
if (c.hashCode() == 144) {
encodingType = true;
return encodingType;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
return encodingType;
}
When you do something like this:
> ctext = text.trim().substring(i, i + 1).toCharArray();
> c = new Character(ctext[0]);
you have to keep in mind that the substring does not work for byte but for characters.
That could produce errors when the original text is ISO codepage, but Java count this as UTF-8. All non-ascii iso characters might be grouped with other bytes (up to 4 bytes) and so your code might not work.
When your original payload is iso codepage, you should first du a conversion to UTF-8
That could be done in just one line of code.
the UTF-8 conversion is happening at Sender JMS channel (Message transformation Bean).
so Java mapping receives the UTF-8 format only....
As per ur input, in ur first reply to this thread...i used it...
-> Is it possible to get the source code of Message transformation Bean to check how it tranform the message from iso to utf8.
because i guess, i m only changing the encoding tag value -- not exactly transforming the payload, as to
relevant <encoding> tag value...
regards,
ashu
Edited by: ashutosh rawat on Nov 22, 2010 1:55 PM
Hi Stefan,
When your XML is in UTF-8, then you cannot compare the characters with their ISO values.
Search is handled with String and Array of characters. in the method fixSpecialCharforWeb
if ((text.trim().substring(i, i + 1).equals("'")) || (c.hashCode() == 8217).
The problem identified : we used a Static field in JavaMapping.
Thanks for your inputs.....
Ashu
User | Count |
---|---|
76 | |
9 | |
8 | |
7 | |
6 | |
5 | |
5 | |
5 | |
5 | |
5 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.