Adding Header in Appended File

[Former Member]

Hello All,

   I have build a scenario SuccessFactors to File. Since i am sending the 200 MB of xml data to the file i have choosen append mode.

   Here i have a challenge of adding header line to appended file.

   I am planning to develop 2 interfaces. one is for appending the file without header line. This is done.

   in Second interface i am planning to add header line to the appended file.

   Can anyone help me to build the second interface to add header line to the output appended file which is present in FTP folder.

   Header will be like below  

  "NOTACTIVE","STUD_ID","FNAME","LNAME","MI","GENDER","JP_ID","JP_DESC","JOB_TITLE","ORG_ID"

   If anyone has any other approach kindly in terms of design kindly let me know.

   I am on PI 7.0 version.

Thanks,

Bhaskar

[Former Member · Accepted solution]

Hello All,

  Can anyone help me with the solution.

Thanks,

Bhaskar

[Emre_tr · Accepted solution]

Hi,

I have not used before but you can try add with parameter in Receiver Channel Field.addHeaderLine .

[Former Member · Accepted solution]

Hi Torun,

   Field.addHeaderLine will add header in every append of the file.

   I need to add header after append of the file is completed. I will create one more interface to add header line but unable to find logic.

Can anyone please help me with development.

Thanks,

Bhaskar

[Former Member · Accepted solution]

Hello All,

   Could anyone please help me with design.

Thanks,

Bhaskar

[anupam_ghosh2 · Accepted solution]

Hi Bhaskar,

                   These are the steps to add header line

1. make two scenarios

2. in the first scenario just append the files.

3. In The second scenario  the sender channel will be on only once a day just before the day ends . Say at 11:58:00 hours , for this use "Availability time planning feature". The channel will stop by 00:00:00 hours next day.

4. In the second scenario add this java mapping code. This code simply adds a header line to the file at the top.

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;




public class RemoveExtraElement  extends AbstractTransformation{






  @Override
  public void transform(TransformationInput arg0, TransformationOutput arg1)
  throws StreamTransformationException {
  // TODO Auto-generated method stub
  execute(arg0.getInputPayload().getInputStream(),arg1.getOutputPayload().getOutputStream());
  }


    public void execute(InputStream in, OutputStream out) throws StreamTransformationException
    {
    try
    {
    StringBuilder s = new StringBuilder();
       byte[] b=new byte[in.available()];
       if(in.read(b) >= 0) {
        s = new StringBuilder(s.toString()+new String(b));
       }

       String header="\"NOTACTIVE\",\"STUD_ID\",\"FNAME\",\"LNAME\",\"MI\",\"GENDER\",\"JP_ID\",\"JP_DESC\",\"JOB_TITLE\",\"ORG_ID\"\n";
       s= new StringBuilder(s.insert(0,header));

       System.out.println(s);
       out.write(s.toString().getBytes());
       in.close();
       out.close();
    }    
    catch(Exception e)
    {
    e.printStackTrace();
    throw new StreamTransformationException(e.getMessage());
    }
    }

}

5. In second  scenario to add java mapping code you need to add dummy Sender and receiver interfaces and do not use FCC anywhere.

Second scenario will be simple file to file scenario.

Let me know if there are any issues.

Hope this resolves the issue.

Regards

Anupam

[Former Member · Accepted solution]

Hi Anupam,

  Thank you for your reply.

  In Second interface in sender channel what would be the file path and file name?

  Do i need to give appended file name and path? or dummy path and file name?

  Please confirm.

  And in header i need to add in the end as !##!

"NOTACTIVE","STUD_ID","FNAME","LNAME","MI","GENDER","JP_ID","JP_DESC","JOB_TITLE","ORG_ID"!##!

So can i declare as below. Please confirm.

"\"NOTACTIVE\",\"STUD_ID\",\"FNAME\",\"LNAME\",\"MI\",\"GENDER\",\"JP_ID\",\"JP_DESC\",\"JOB_TITLE\",\"ORG_ID\"!##!\n";

Thanks,

Bhaskar

[justin_santhanam · Accepted solution]

Hi Bhaskar,

Is your file name going to be static all the time (I believe so yes, so that you can append it) and where are you writing this file is it in NFS or FTP?

I'm thinking of using below options.

- Create a custom adapter module( simple one_

- Just check if the File you are going to write already exists in the specified folder

- If it exists don't do anything just pass your payload,

- If it doesn't then add header

- Add this module in the Receiver File adapter

So everytime when the File is gong to write, it will check for the file (name exists in the folder) , if it exists it won't write the header, if not it will write the header.

Just a thought!

Justin.     

[Former Member · Accepted solution]

Hi Justin,

  First interface will append the file and place it in NFS folder. Second interface will pick the appended file and with java mappings as per Anupam code with dummy interface [its like pass through] will send it to the FTP folder with header added.

  First interface file name will created as XXX_datetimestamp.csv

  So now i got the idea that in second interface sender channel i need to put the XXX_*.csv and send it to FTP server. Java mapping will take care of adding header line to the file.

Thanks,

Bhaskar

[justin_santhanam · Accepted solution]

Okay cool!

Thnx.

[Former Member · Accepted solution]

Hi Justin/Anupam,

  Like i add header i need to add footer as well. Footer will be as below

Footer like:  TB TA TC first date last date of the month record count

      For Ex:   TB TA TC 2015/12/01 2015/12/31 3000

In few interfaces i have only header and in few footer and in few header and footer.

If it is only footer how can i achieve and if its both header and footer how can i achieve.

Please help me with design.

Thanks,

Bhaskar

[anupam_ghosh2 · Accepted solution]

Hi Bhaskar,

your query

-----------------

1. "In Second interface in sender channel what would be the file path and file name?

  Do i need to give appended file name and path? or dummy path and file name?"

ans) You need to give the name of the appended file and target directory of the receiver channel you used in first scenario.

2.

And in header i need to add in the end as !##!

"NOTACTIVE","STUD_ID","FNAME","LNAME","MI","GENDER","JP_ID","JP_DESC","JOB_TITLE","ORG_ID"!##!

So can i declare as below. Please confirm.

"\"NOTACTIVE\",\"STUD_ID\",\"FNAME\",\"LNAME\",\"MI\",\"GENDER\",\"JP_ID\",\"JP_DESC\",\"JOB_TITLE\",\"ORG_ID\"!##!\n";

ans) This is correct.

Regards

Anupam

[anupam_ghosh2 · Accepted solution]

Hi Bhaskar,

                   Try this mapping for including both header and footer. You can add and remove lines of code to include the footer or remove it.

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;




public class RemoveExtraElement  extends AbstractTransformation{


  /**
  * @param args
  * @throws FileNotFoundException 
  * @throws StreamTransformationException 
  */

  @Override
  public void transform(TransformationInput arg0, TransformationOutput arg1)
  throws StreamTransformationException {
  // TODO Auto-generated method stub
  execute(arg0.getInputPayload().getInputStream(),arg1.getOutputPayload().getOutputStream());
  }
    public void execute(InputStream in, OutputStream out) throws StreamTransformationException
    {
    try
    {
    StringBuilder s = new StringBuilder();
       byte[] b=new byte[in.available()];
       if(in.read(b) >= 0) {
        s = new StringBuilder(s.toString()+new String(b));
       }

       String header="\"NOTACTIVE\",\"STUD_ID\",\"FNAME\",\"LNAME\",\"MI\",\"GENDER\",\"JP_ID\",\"JP_DESC\",\"JOB_TITLE\",\"ORG_ID\"!##!\n";
       s= new StringBuilder(s.insert(0,header));
       String footer="\nTB TA TC 2015/12/01 2015/12/31 3000";
       s=s.append(footer);
       System.out.println(s);
       out.write(s.toString().getBytes("UTF-8"));
       in.close();
       out.close();
    }    
    catch(Exception e)
    {
    e.printStackTrace();
    throw new StreamTransformationException(e.getMessage());
    }
    }

}

Regards

Anupam

[Former Member · Accepted solution]

Hi Anupam,

  Thank you for your reply.

  1. In footer i need to get date dynamically and number of records count. In my above question i have given 3000 as a count but it varies according to the record count in the file.

    for ex: In Dec it should be like below.

              TB TA TC 2015/12/01 2015/12/31 3000 [record count of the appended file.]

    For Ex: In Jan it should be like below.

              TB TA TC 2016/01/01 2016/01/29 4256 [record count of the appended file.]

      2105/12/01 and 2016/01/01 is the first date of the interface run month.

      2015/12/31 and  2016/01/29 last date of the interface run month. [excluded weekend for Jan]

     Dates will vary. Suppose of interface run start date is 4th Jan 2016 [2015/01/04] then it should come as it is instead 2016/01/01. same for last date of interface run month.

last date for Jan 2016 is 29th [taken only last working day] so date should come as 2016/01/29.

      3000 and 4256 is the record count of appended file.

  I am hoping this description will help you....

Thanks you,

Bhaskar

[Former Member · Accepted solution]

Hi Anupam,

  Thank you for your reply.

  1. In footer i need to get date dynamically and number of records count. In my above question i have given 3000 as a count but it varies according to the record count in the file.

    for ex: In Dec it should be like below.

              TB TA TC 2015/12/01 2015/12/31 3000 [record count of the appended file.]

    For Ex: In Jan it should be like below.

              TB TA TC 2016/01/01 2016/01/29 4256 [record count of the appended file.]

      2105/12/01 and 2016/01/01 is the first date of the interface run month.

      2015/12/31 and  2016/01/29 last date of the interface run month. [excluded weekend for Jan]

     Dates will vary. Suppose of interface run start date is 4th Jan 2016 [2015/01/04] then it should come as it is instead 2016/01/01. same for last date of interface run month.

last date for Jan 2016 is 29th [taken only last working day] so date should come as 2016/01/29.

      3000 and 4256 is the record count of appended file.

  I am hoping this description will help you....

Thanks you,

Bhaskar

[anupam_ghosh2 · Accepted solution]

Hi Bhaskar,

                    Within PI server, I can get the current date, i.e the last time when the interface executed. How will PI server know when did this interface ran for the first time in the month??

Record count stuff can be added easily.

Regards

Anupam

[Former Member · Accepted solution]

Hi Anupam,

  First working day of the month would be the first interface run date.

  Let us make simpler...let us make every month 1st would be the start date of interface run month.

  and calender last date will make last date of the interface run month for ex: 31st in dec and 29th in feb.

  Hope this will make easier.

  How can we get the record count. Could you please help me.

Thanks,

Bhaskar

[anupam_ghosh2 · Accepted solution]

Hi Bhaskar,

                  Suppose on 15th of jan a file is created. What will be the end date in the footer of this file?

Newline count will give you the number of records in the file. 

Regards

Anupam

[Former Member · Accepted solution]

Hi Anupam,

  As per FD, the interface will run on 1st of every month.

  Eligibility start date: 1st of every month.

  Eligibility end date: last date of the month.

  This is what they have given in spec.

Thanks,

Bhaskar

[anupam_ghosh2 · Accepted solution]

Hi Bhaskar,

                  It has taken a lot of time to answer this query as it involved lot of trial and error. Try this code. I have assumed that last line of original file has no newline at the end. In case it has just initialise value of count variable to -1 instead of zero.

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Calendar;
import java.util.Date;


import com.sap.aii.mapping.api.AbstractTransformation;
import com.sap.aii.mapping.api.StreamTransformationException;
import com.sap.aii.mapping.api.TransformationInput;
import com.sap.aii.mapping.api.TransformationOutput;




public class RemoveExtraElement  extends AbstractTransformation{


  /**
  * @param args
  * @throws FileNotFoundException
  * @throws StreamTransformationException
  */



  @Override
  public void transform(TransformationInput arg0, TransformationOutput arg1)
  throws StreamTransformationException {
  // TODO Auto-generated method stub
  execute(arg0.getInputPayload().getInputStream(),arg1.getOutputPayload().getOutputStream());
  }
    public void execute(InputStream in, OutputStream out) throws StreamTransformationException
    {
    try
    {
    StringBuilder s = new StringBuilder();
       byte[] b=new byte[in.available()];
       if(in.read(b) >= 0) {
        s = new StringBuilder(s.toString()+new String(b));
       }

       String header="\"NOTACTIVE\",\"STUD_ID\",\"FNAME\",\"LNAME\",\"MI\",\"GENDER\",\"JP_ID\",\"JP_DESC\",\"JOB_TITLE\",\"ORG_ID\"!##!\n";
       s= new StringBuilder(s.insert(0,header));
       java.util.Date date= new Date();
       Calendar cal = Calendar.getInstance();
       cal.setTime(date);

       int month = cal.get(Calendar.MONTH)+1;

       int year=cal.get(Calendar.YEAR);
       int day=cal.get(Calendar.DAY_OF_MONTH);
       cal.set(Calendar.YEAR,year);
       cal.set(Calendar.MONTH,month);
       cal.set(Calendar.DATE,day);
       String runDate1=""+year+"/";
       if(month<10)
       {
        runDate1=runDate1+"0";
       }
       runDate1=runDate1+month+"/";

       String runDate2=runDate1+cal.getActualMaximum(Calendar.DAY_OF_MONTH);
       runDate1+="01";

       int len=s.length();
       int count=0;
       for(int i=0;i<len;++i)
       {
                  if(s.charAt(i)=='\n')
                  {
                 count++;
                  }
       }
       String footer="\nTB TA TC "+runDate1+" "+runDate2+" "+count;
       s=s.append(footer);
       System.out.println(s);
       out.write(s.toString().getBytes("UTF-8"));
       in.close();
       out.close();
    }   
    catch(Exception e)
    {
    e.printStackTrace();
    throw new StreamTransformationException(e.getMessage());
    }
    }

}

Regards

Anupam

[justin_santhanam]

Bhaskar.

I'm thinking the below design will work for Header only files. Still thinking through the Files that has footers. But I don't like the idea of bringing in the whole file back into PI , which later  might lead to memory/heap issues.

Anupam, has given a perfect solution, but I'm just only concerned with the volume of file. Think through this and see if it make sense.

@Anupam and/all experts- Please feel free to chip in and let me know if there is any flaws you can see in the above design.

Thank you,

Justin.

[anupam_ghosh2]

Hi Justin ,

          If you only consider the header line this seems to be a very good solution. This will avoid use of java code.

Regards'

Anupam

[justin_santhanam]

Anupam,

Just need a clarification. When you say using Java mapping we can get the record count, are you thinking of bringing the whole 200 MB (less or more) file back into PI through the second  interface and do the record count?

Thanks,

Justin.

[anupam_ghosh2]

Hi Justin,

               Second interface is already reading the file. I am just counting the newlines except the header. You can see the code I posted here.

Regards

Anupam

[Former Member]

Thank you Anupam for your help.

Thank you Justin for your helpful suggestions.

I have one more discussion of just appending footer with record count. Could you please help.

link: 

Thanks,

Bhaskar

[anupam_ghosh2]

Thank you bhaskar for providing the update and marking my answer as correct answer.

Thank you again for posting a question like this. While providing a solution, I learnt a lot of new things.

Special thanks to Justin for providing all the valuable inputs to this thread. All your posts are very helpful indeed.

Regards

Anupam.