Loading data having large number of columns

[Former Member]

Hi,

I have a table having more than 700 columns and 1000000 rows. It is a transactional table. While doing the Initial load though BODS, to Sybase IQ, it took about 72 hours ,to load just 84000 records. Is there a workaround to get this loaded faster?

Thank you,

Priyanka

[Former Member]

700 columns in table..is that even possible, how you managed to map all 700.

[former_member187605]

Sandeep Pandey wrote:

700 columns in table..is that even possible, how you managed to map all 700.

1-to-1 mapping and a template table?

My first question: do you need all that data in IQ? Data load time is related to the volume of data = #records * #columns * avg. column size. Removing unneccesary columns (and records!) will reduce the load time.

Further questions: what's the source system like? anything pushed down to db? any network latency? what's the DS server sizing? have you been able to identify the bottleneck (extract, transform or load) in the process? are you using bulk load? what's the DOP? when not 1-to-1 are there complex transformations?

There are dozens of parameters that might impact performance, so a little bit extra information about your application might come in very useful.