MacRumors Forums speed of memory allocation
 User Name Remember Me? Password

 Thread Tools Search this Thread Display Modes
 Aug 10, 2013, 07:31 PM #1 farmerdoug macrumors 6502a   Join Date: Sep 2008 speed of memory allocation Once number_of_inidices gets over 40, things get very slow. number_of_buy_channels = number_of_sell_channels ~ 10 days_of_data = 3400 Suggestions? thanks Code: ```signal = (int******) calloc (number_of_indices, sizeof(int*****)); for (i = 0; i < number_of_indices; i++) { if( (signal[i] = (int*****) calloc (number_of_indices, sizeof(int ****))) == NULL) printf("couldn't allocated memory\n"); for (j = 0; j < number_of_indices; j++) { if ((signal[i][j] = (int****) calloc (num_buy_channels, sizeof(int ***))) == NULL) printf("couldn't allocated memory\n"); for (k = 0; k < num_buy_channels; k++) {//printf("c"); if ((signal[i][j][k] = (int***) calloc (num_sell_channels, sizeof(int **))) == NULL) printf("couldn't allocated memory\n"); for (l = 0; l < num_sell_channels; l++) {//printf("d\n"); if ((signal[i][j][k][l] = (int**) calloc(2, sizeof(int *))) == NULL) printf("couldn't allocated memory\n"); for (m = 0; m < 2; m++) // number of fk values if ( (signal[i][j][k][l][m] = (int*) calloc(days_of_data, sizeof(int ))) == NULL) printf("couldn't allocated memory\n"); } } } }``` 0
 Aug 10, 2013, 07:35 PM #2 robvas macrumors 68000   Join Date: Mar 2009 Location: USA Open up Activity monitor and check if you are running out of physical memory and are starting to use virtual memory. That would really slow things down. 0
 Aug 10, 2013, 07:46 PM #3 farmerdoug Thread Starter macrumors 6502a   Join Date: Sep 2008 Not sure what I see when I open the Activity Monitor. I'll ask my kid when I see him. Thanks. 0
 Aug 10, 2013, 07:54 PM #4 ElectricSheep macrumors 6502   Join Date: Feb 2004 Location: Wilmington, DE Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc(). Unless you actually need to manage any of those allocations, I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension. __________________ 15'' MBP (early 2011) | i7 3770k Hackintosh | i7 Mac Mini (late 2012) | iPhone 5 | iPad 3 (2012) | iPad mini | MacOS X 10.9.2 0
 Aug 10, 2013, 07:59 PM #5 farmerdoug Thread Starter macrumors 6502a   Join Date: Sep 2008 That seems to work although I now get an error later when signal ( and other things are sent to a procedure). That's for tomorrow. Thanks 0
 Aug 10, 2013, 10:35 PM #6 robvas macrumors 68000   Join Date: Mar 2009 Location: USA Using Activity Monitor to read System Memory and determine how much RAM is being used sizeof(int*****) Seriously? 0
 Aug 10, 2013, 11:01 PM #7 lee1210 macrumors 68040     Join Date: Jan 2005 Location: Dallas, TX Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant. It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans. -Lee 0
Aug 11, 2013, 09:26 AM   #8
subsonix
macrumors 68040

Join Date: Feb 2008
Quote:
 Originally Posted by ElectricSheep I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.
No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.

----------

Quote:
 Originally Posted by ElectricSheep Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc().
Yep nuts, the OS has to keep track of 320,000 pointers for 1 process alone for what is essentially one object.
0
Aug 11, 2013, 11:42 AM   #9
gnasher729
macrumors G5

Join Date: Nov 2005
Quote:
 Originally Posted by lee1210 Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant. It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans. -Lee
So the solution is:

1. Upgrade RAM. 8 GB for number_of_indices < 50, 16 GB for number_of_indices < 70.
2. If possible, store "short" instead of int; that gets you twice as far.
3. Change the algorithm. Whatever the algorithm is.

I would be quite surprised if these billions of array elements each contained actually useful data.
0
Aug 11, 2013, 12:11 PM   #10
ElectricSheep
macrumors 6502

Join Date: Feb 2004
Location: Wilmington, DE
Quote:
 Originally Posted by subsonix No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.
That occurred to me after I posted, but I was rather tired. It is still possible to perform a single large allocation, and manually construct the multidimensional array.

Realistically, it seems like this application might benefit from a relational model, especially if the data is sparse.
__________________
15'' MBP (early 2011) | i7 3770k Hackintosh | i7 Mac Mini (late 2012) | iPhone 5 | iPad 3 (2012) | iPad mini | MacOS X 10.9.2

Last edited by maflynn; Aug 11, 2013 at 03:05 PM. Reason: Fixed quote tag
0
 Aug 11, 2013, 06:35 PM #11 farmerdoug Thread Starter macrumors 6502a   Join Date: Sep 2008 1. sizeof(int*****) yes, it helps me keep track of where I am. 2. Yes every element in the array is filled. 3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits. 4. Time to rethink. 0
 Aug 11, 2013, 07:59 PM #12 lloyddean macrumors 6502a   Join Date: May 2009 Location: Des Moines, WA What computer, OS version, hard drive free space and amount of RAM is this running on? 0
Aug 12, 2013, 02:47 AM   #13
gnasher729
macrumors G5

Join Date: Nov 2005
Quote:
 Originally Posted by farmerdoug 1. sizeof(int*****) yes, it helps me keep track of where I am. 2. Yes every element in the array is filled. 3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits. 4. Time to rethink.
I don't doubt that every array element is filled, but mostly with calculated values that don't need to be stored.

Quote:
 Originally Posted by lloyddean What computer, OS version, hard drive free space and amount of RAM is this running on?
The first three are quite irrelevant; the fourth one is "not enough".
0
 Aug 12, 2013, 10:42 AM #14 lloyddean macrumors 6502a   Join Date: May 2009 Location: Des Moines, WA Bad response. If you had all the answers you wouldn't be here. 0
Aug 12, 2013, 11:00 AM   #15
robvas
macrumors 68000

Join Date: Mar 2009
Location: USA
Quote:
 Originally Posted by farmerdoug 3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
Computer error at ______ crashes market.
0

 MacRumors Forums

 Thread Tools Search this Thread Search this Thread: Advanced Search Display Modes Linear Mode

 Similar Threads thread Thread Starter Forum Replies Last Post adamneer OS X Mavericks (10.9) 3 Oct 26, 2013 05:11 PM cgm707 MacBook Pro 12 Jun 23, 2013 06:28 AM Will G. Mac Pro 1 Feb 9, 2013 07:47 PM michial MacBook Pro 3 Nov 28, 2012 10:06 AM goinskiing MacBook Air 38 Nov 1, 2012 09:48 AM