ESP project memory leaks : diagnosing

[Former Member]

Hello,

I have an ESP project that grows in its memory footprint far beyond what I expect and/or can tolerate.

I suspect I am not managing memory properly in my FLEX components (I have two of them, where most

of my heavy lifting is being done, each with several input windows).

Where does one begin to identify where the leaks are?

Thanks,

Michael

[Former Member · Accepted solution]

Hi,

Flex code is difficult to debug and the most usefull statement might be the print statement.

Is there any reason that you suspect the Flex components for the memory leak?

Do you for example see a large difference if you comment out all flex code and ouput fixed or random data from the flex?

What version of ESP are you exactly using, do you have the version string?

Without knowing the flex code.

One problem with memory was, I believe, forgetting to reset vectors.

If you have a vector then you always have to explictly resize it back to zero, always, like at the end before you re-use it.

vector(...) v_stack;

..

resize(v_stack, 0);

With higher debug levels (7 or higher) you could see some memory usage details in the log file, but you might not see much of your flex. But you can't do this with high load because the log file will explode.

For example run the project a while and then stop the input.

With the esp_client connect to your project and add to the commandline the following to enable logging: "logelevel 7"

Add some manual input that will be processed by your flex and afterwards stop the project.

The esp_server.log file should now contain some information on queue sizes and memory data that you can analyze.

Hopefully this will give some direction.

Thanks,

Ben

[Former Member · Accepted solution]

Hi Ben,

First of all, thanks for the response.  Second, I no longer suspect anything special regarding my use (or misuse) of Flex.  The behavior I'm seeing (using top -b -p <pid> on a Unix command-line) is evident in a brand new, very simple project with no Flex at all.  I have 4 input windows with a primary key defined for each, and an ALL retention policy.  Each window is tied to a schema defined globally, and is bound to an input adapter of type "db_in".

What I am witnessing is a steady growth of both VIRT and RES memory for the pid associated with my ESP project.

My question is why?  I gather that I am not grasping something fundamental about ESP.  With windows, having a primary key......I assumed they would fill up, reach a steady state, and then keep my project memory footprint constant.  I guess not.  What am I not doing?

We're running ESP 5.1 SP04 on RHEL 6  (studio : Version: 5.1.04.00/20131119.1/SP04 PL00/winnt/x86_64/64-bit/OPT/Tue, Nov 19, 2013  4:06:09 AM) (server : ESP 5.1.04.00).

Cheers,

Michael

[Former Member · Accepted solution]

Quickly following up on my last post.  After explicitly setting the "memory" option in my project configuration file (.ccr) to "2048" (megabytes) and redeploying my project, and watching the same accumulation of memory.....the cluster stopped and restarted my project (breaking connections to ESP Studio where I am watching the contents of the windows) precisely when the VIRTUAL memory hit 2048.

Have you seen this before?

Michael

[former_member217348 · Accepted solution]

Hi Michael,

I noticed that you are using ESP 5.1 SP04. There was a memory leak bug that got fixed in ESP 5.1 SP04 PL03. Please try upgrading to at least that version of ESP, if not to the latest version which is ESP 5.1 SP08 PL01.

Thank you,

Alice

[Former Member · Accepted solution]

Michael,

From your description I can't say how many windows you are sending to the Window.

With your simple model where you have 4 input Windows with a KEEP ALL policy the only way that I can see the memory continuously growing is when you have only inserts flowing into the Windows.  Can you try set a retention policy say KEEP 100 ROWS and see if memory still grows?

[Former Member · Accepted solution]

Alice,

I upgraded last evening from SP04 to SP04 PL03.  The leaks are history.  For a patch installation

it was a little bumpy (I had to restore service.xml and keystore_rsa.jks from my backup after the installation), with questions that should not be asked (like naming the cluster, setting passwords, etc)

but those potholes were mentioned in the cover letter for EBF 23062.

Turns out we are using the generic database adapter (ODBC) extensively to poll our sources and that seems to have gotten right to the heart of the leak we were witnessing.

Thanks again,

Michael

Message was edited by: Michael Scharber (corrected target patch level).