PDA

View Full Version : speed of memory allocation




farmerdoug
Aug 10, 2013, 07:31 PM
Once number_of_inidices gets over 40, things get very slow.
number_of_buy_channels = number_of_sell_channels ~ 10
days_of_data = 3400

Suggestions?
thanks


signal = (int******) calloc (number_of_indices, sizeof(int*****));
for (i = 0; i < number_of_indices; i++)
{
if( (signal[i] = (int*****) calloc (number_of_indices, sizeof(int ****))) == NULL)
printf("couldn't allocated memory\n");
for (j = 0; j < number_of_indices; j++)
{
if ((signal[i][j] = (int****) calloc (num_buy_channels, sizeof(int ***))) == NULL)
printf("couldn't allocated memory\n");
for (k = 0; k < num_buy_channels; k++)
{//printf("c");
if ((signal[i][j][k] = (int***) calloc (num_sell_channels, sizeof(int **))) == NULL)
printf("couldn't allocated memory\n");
for (l = 0; l < num_sell_channels; l++)
{//printf("d\n");
if ((signal[i][j][k][l] = (int**) calloc(2, sizeof(int *))) == NULL)
printf("couldn't allocated memory\n");
for (m = 0; m < 2; m++) // number of fk values
if ( (signal[i][j][k][l][m] = (int*) calloc(days_of_data, sizeof(int ))) == NULL)
printf("couldn't allocated memory\n");
}
}
}
}



robvas
Aug 10, 2013, 07:35 PM
Open up Activity monitor and check if you are running out of physical memory and are starting to use virtual memory. That would really slow things down.

farmerdoug
Aug 10, 2013, 07:46 PM
Not sure what I see when I open the Activity Monitor. I'll ask my kid when I see him.
Thanks.

ElectricSheep
Aug 10, 2013, 07:54 PM
Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc(). Unless you actually need to manage any of those allocations, I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.

farmerdoug
Aug 10, 2013, 07:59 PM
That seems to work although I now get an error later when signal ( and other things are sent to a procedure). That's for tomorrow.
Thanks

robvas
Aug 10, 2013, 10:35 PM
Using Activity Monitor to read System Memory and determine how much RAM is being used (http://support.apple.com/kb/HT1342)

sizeof(int*****)

Seriously?

lee1210
Aug 10, 2013, 11:01 PM
Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

-Lee

subsonix
Aug 11, 2013, 09:26 AM
I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.

No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.

----------

Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc().

Yep nuts, the OS has to keep track of 320,000 pointers for 1 process alone for what is essentially one object.

gnasher729
Aug 11, 2013, 11:42 AM
Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

-Lee

So the solution is:

1. Upgrade RAM. 8 GB for number_of_indices < 50, 16 GB for number_of_indices < 70.
2. If possible, store "short" instead of int; that gets you twice as far.
3. Change the algorithm. Whatever the algorithm is.

I would be quite surprised if these billions of array elements each contained actually useful data.

ElectricSheep
Aug 11, 2013, 12:11 PM
No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.

That occurred to me after I posted, but I was rather tired. It is still possible to perform a single large allocation, and manually construct the multidimensional array.

Realistically, it seems like this application might benefit from a relational model, especially if the data is sparse.

farmerdoug
Aug 11, 2013, 06:35 PM
1. sizeof(int*****)
yes, it helps me keep track of where I am.
2. Yes every element in the array is filled.
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
4. Time to rethink.

lloyddean
Aug 11, 2013, 07:59 PM
What computer, OS version, hard drive free space and amount of RAM is this running on?

gnasher729
Aug 12, 2013, 02:47 AM
1. sizeof(int*****)
yes, it helps me keep track of where I am.
2. Yes every element in the array is filled.
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
4. Time to rethink.

I don't doubt that every array element is filled, but mostly with calculated values that don't need to be stored.

What computer, OS version, hard drive free space and amount of RAM is this running on?

The first three are quite irrelevant; the fourth one is "not enough".

lloyddean
Aug 12, 2013, 10:42 AM
Bad response.

If you had all the answers you wouldn't be here.

robvas
Aug 12, 2013, 11:00 AM
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.

Computer error at ______ crashes market.