cancel
Showing results for 
Search instead for 
Did you mean: 

file_csv_input : adapter breaking following 5.1 SP04 PL03 upgrade

Former Member
0 Kudos

Folks,

Has anyone experienced the breakage of both file_csv_input and file_csv_output adapters following an upgrade to a higher patch

level?  This strikes me as a fairly simple interface, yet my project components using the adapters no longer function following the patch.

Looking for a place to start.

Thanks,

Michael

Accepted Solutions (0)

Answers (1)

Answers (1)

Former Member
0 Kudos

Hello,

I'm not clear on which adapters you are using.  Are you using the "toolkit_file_csv_input" adapters?

   File/Hadoop CSV Input and Output Adapter

If so, I would look for two log files in your project's working/run-time directory:

  • "esp_server.log" will show us any errors specific to the project
  • "frameworkadapter.log" will show us adapter specific errors

File and Directory Infrastructure

Thanks,

Neal

Former Member
0 Kudos

Hi Neal,

Apologies.  I am indeed using the toolkit_file_csv_input and toolkit_file_csv_output adapters.

The relevant fragment from esp_server.log is the following :

2014-10-07 02:20:59.275 | 28649 | container | [SP-3-148005] (1.093) sp(28521) ConnectionReader(IndexBetaIN) error reading from Connection(IndexBetaIN_CSV): 2014-10-07 02:20:59 The execution command "/local/sw/esp5.1/ESP-5_1/adapters/framework/bin/start.sh" "/local/sw/esp5.1/ESP-5_1/adapters/framework/instances/file_csv_input/adapter_config.xml" "-DFileInputTransporterParameters.Dir=/local/working/data/iv" "-DFileInputTransporterParameters.File=IndexBeta.csv" "-DFileInputTransporterParameters.DynamicMode=" "-DFileInputTransporterParameters.PollingPeriod=600" "-DFileInputTransporterParameters.RemoveAfterProcess=false" "-DFileInputTransporterParameters.ScanDepth=" "-DCsvStringToEspFormatterParameters.Delimiter=," "-DCsvStringToEspFormatterParameters.DateFormat=YYYYMMDD" "-DCsvStringToEspFormatterParameters.TimestampFormat=" "-DCsvStringToEspFormatterParameters.HasHeader=true" "-DCsvStringToEspFormatterParameters.ExpectStreamNameOpcode=false" returned code: 127. 0 records consumed, 0 read, 0 bad, 0 good.

2014-10-07 02:20:59.275 | 28649 | container | [SP-4-148003] (1.093) sp(28521) ConnectionReader(IndexBetaIN) exiting.

frameworkadapter.log  is not being written to....for my project, but used to be (prior to the patch).


Does this shed any light?


Greatly appreciated (as usual).


Cheers,


Michael

Former Member
0 Kudos

Hello,

It looks like it is trying to start the adapter with the "/local/sw/esp5.1/ESP-5_1/adapters/framework/bin/start.sh" script but it fails with error code 127.  If it is unable to start, nothing will be written to the "frameworkadapter.log".

If you "ls -l $ESP_HOME/adapters/framework/bin/start.sh", does the user who started the ESP cluster manager have permission to read & execute this file?

Thanks,

Neal

Former Member
0 Kudos

Hi Neal,

The user does have rights to read/execute that script.  I ran the command in it's entirety from the command-line and it does cough up more information :

10-07-2014 19:04:56.955 INFO [main] (Framework.main) start /local/sw/esp5.1/ESP-5_1/adapters/framework/instances/file_csv_input/adapter_config.xml

10-07-2014 19:04:57.471 INFO [main] (XmlUtils$1.resolveResource) /local/sw/esp5.1/ESP-5_1/adapters/framework/config/parametersdefine.xsd

10-07-2014 19:04:57.536 INFO [main] (XmlUtils$1.resolveResource) /local/sw/esp5.1/ESP-5_1/adapters/framework/config/standard_module_parametersdefine.xsd

10-07-2014 19:04:57.944 INFO [main] (PortAllocator.allocateFromDataCenter) Trying to get global lock ...

10-07-2014 19:04:57.945 INFO [main] (PortAllocator.allocateFromDataCenter) Success to get global lock.

10-07-2014 19:04:58.188 INFO [main] (AdapterController.init) Address 45.32.82.94:19082 is used to accept the control command.

10-07-2014 19:04:58.225 INFO [main] (AdapterController.sendCommand) start /local/sw/esp5.1/ESP-5_1/adapters/framework/instances/file_csv_input/adapter_config.xml

10-07-2014 19:04:58.405 INFO [Thread-0] (Server.doStart) jetty-7.6.1.v20120215

10-07-2014 19:04:58.441 INFO [Thread-0] (ContextHandler.startContext) started o.e.j.s.ServletContextHandler{/,null}

10-07-2014 19:04:58.489 INFO [Thread-0] (AbstractConnector.doStart) Started SelectChannelConnector@45.32.82.94:19082

10-07-2014 19:04:59.404 INFO [main] (AdapterController.executeStart) Adapter controller is started.

10-07-2014 19:04:59.404 INFO [main] (AdapterController.executeStart) Starting adapter

10-07-2014 19:04:59.435 INFO [main] (ModuleWrapper.initQue) Buffer Size for module MyInStream_Publisher is 10240.

10-07-2014 19:04:59.436 INFO [main] (ModuleWrapper.initParallelParameters) Parallel setting of module MyInStream_Publisher is true.

10-07-2014 19:04:59.436 INFO [main] (EspPublisher.init) EspPublisher is initializing

10-07-2014 19:04:59.438 ERROR [main] (EspConnectorWrapper.init) Error code:400842, Severity : 3 (Error)

Error message:The stream name is not configured.

Error description:The stream name is not configured.

10-07-2014 19:04:59.441 INFO [main] (ContextHandler.doStop) stopped o.e.j.s.ServletContextHandler{/,null}

Former Member
0 Kudos

Hello,

OK, those errors would be expected because if you try to run the adapter in unmanaged mode (manually outside of a project), you have to edit the "/local/sw/esp5.1/ESP-5_1/adapters/framework/instances/file_csv_input/adapter_config.xml" file and uncomment two lines:

  <!--Uncomment the following 2 elements when you use 'start_adapter[.bat|.sh]' -->

          <!--ProjectName>EspProject1</ProjectName-->

          <!--StreamName>BaseInput</StreamName-->

And also fill in the server/project URI, username and password:

      <Uri>esp://localhost:19011/sample_workspace/file_csv_input</Uri>

      <Security>

        <User></User>

        <Password encrypted="false"></Password>

Thanks,

Neal

Former Member
0 Kudos

p.s.

You would leave "ProjectName=EspProject1" and change "Streamname=" to the stream/window you want it to publish to.

You could go ahead and try this and see if it gets further and raises any additional errors.  If no errors, make sure to change everything back to the way it was.

Some other things to try/check:

1) Make sure there are no stray adapter processes.  "ps -ef | grep java" and see if there are any running out of the ESP directory path.

2) You could get an strace while trying to start the adapter...

   a) You could do a "ps -ef | grep esp_server" and look for the PID of the esp_server process belonging to your project.  It would have some command line arguments similar to "--cluster-container --cluster-container-id Id_107_1412629980721".

   b) strace -f -o esp_strace.out -p <esp_server_PID>

   c) In ESP Studio you can expand the plus sign next to the window/stream the adapter is attached to and then right click the adapter and try starting it.

   d) Send us the "esp_strace.out".

Former Member
0 Kudos

Neal,

From the command-line, following your advice on changing (for testing purposes only) adapter_config.xml, the contents of my CSV file were successfully published into the receiving window of my running ESP project!

Now what?   Wondering if it isn't just a simple configuration pertaining to the adapter that got muffed.

What else do you need from me?   Very eager to get this resolved and feels like we are very close.

Cheers,

Michael

Former Member
0 Kudos

Hello,

There weren't any stray/hung esp_server or java processes running at the time of upgrade?  Or when you run the project?  I have seen upgrades fail when there were running processes on Windows because the installer couldn't overwrite everything it needed to.  I haven't seen the same thing happen on Linux but I thought I should raise the question.

Can you try the "strace" instructions?  In the strace, I would look for where it tries to execute "start.sh" and follow on from there looking for errors.

If you don't feel comfortable posting that here, could you open a technical support case and attach the diagnostics to it?

Thanks,

Neal

Former Member
0 Kudos

Hi Neal,

I have confirmed from my notes, and my command history that (as I normally do on any ESP full recycle procedure) I confirmed no orphaned or running processes associated with "esp".

Also recall that toolkit_file_csv_output is also no longer functional.

I'll continue to troubleshoot a bit and then look for alternatives to my CSV in/out problem.

Thanks,

Michael

Former Member
0 Kudos

Hi Michael,

Yes, I understand that the adapter is no longer functional.  The "strace" diagnostics I suggested getting could really shed a lot of light on the problem.

As long as the project itself can run, you can start "strace" on the esp_server PID belonging to the product.

Once the strace is running, you can go into Studio and right click on the adapter (see the red arrow in the picture) and tell it to start:

The strace then would capture the start-up process and what went wrong.

Thanks,

Neal

Former Member
0 Kudos

Hi Neal,

I had these CSV adapters in a big, complicated (to me) project....so it dawned on me to make a brand new project (from Studio) and put just one of these into it (the input version) and see what happens.  I did this from Studio, deployed from Studio, started from Studio and it worked fine, to my amazement.

Then, I took the "stock" .ccr file generated by Studio (and which got deployed to the server) and replaced the .ccr file I had been using for months with my other project, with that one.   It brought

the two instances of my CSV adapters back to life.  So now I am trying to isolate exactly which configuration parameter in my real project is suddenly causing all of this strife for the CSV adapters.

I should have this information shortly.

Cheers,

Michael

Former Member
0 Kudos

Neal,

This project configuration option is causing my toolkit_file_csv_input and toolkit_file_csv_output adapters to hang/fail :


     <Option name="memory" value="4096"/>


This is from the <Configuration><Deployment><Project><Options> hierarchy.  I had set it last week

after battling the memory leak caused by the generic db input adapters....to protect my host from being eaten alive memory-wise.  Now that the leak(s) are gone (one hopes) I suppose I could comment this option out

but it seems backwards to have to do so.  At least, maybe you can use this information to test on your end

with a simple project having one instance of a toolkit_file_csv_input adapter.....and set this option in your

project deployment...and see what happens.


Let me know what you think.


Cheers,


Michael

Former Member
0 Kudos

Hello,

How much memory does the adapter use when you run it on your server?  I set the memory value to 4096 in my CCR and ran the project but the adapter still started.

Setting the memory option is essentially setting the ulimit/limit for memory use to prevent an application from using more than that limit.  If it tries to malloc() more than what it is allowed (or what is available), malloc() will fail with ENOMEM.

Is your CSV file large?  Are there a large number of columns in the file (and associated window/stream)?

For my simple test I don't see the "java" adapter process using very much memory:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                             

4623 nstack    20   0 1674m  87m  13m S    2  0.4   0:04.98 java                                                                                                                                                                                            

4549 nstack    20   0  841m  50m  23m S    0  0.2   0:00.38 esp_server

If I set the memory setting below 1674 MB, the adapter fails to start:

   <Option name="memory" value="1500"/>

I would consider this expected behavior.  If the adapter can't get enough memory to do it's job, it will fail.  If esp_server calls malloc() and gets ENOMEM, it will fail (most likely crash).

The "memory" setting is not like setting a pool and telling ESP to use only what is in the pool.  It is like setting a roof and when ESP hits the roof, bad things happen.

Thanks,

  Neal