speed of memory allocation

Discussion in 'Mac Programming' started by farmerdoug, Aug 10, 2013.

  1. macrumors 6502a

    Joined:
    Sep 16, 2008
    #1
    Once number_of_inidices gets over 40, things get very slow.
    number_of_buy_channels = number_of_sell_channels ~ 10
    days_of_data = 3400

    Suggestions?
    thanks

    Code:
    signal = (int******) calloc (number_of_indices, sizeof(int*****));
    		for (i = 0; i < number_of_indices; i++)
    			{
    			if( (signal[i] = (int*****) calloc (number_of_indices, sizeof(int ****))) == NULL)
    				printf("couldn't allocated memory\n");
    			for (j = 0; j < number_of_indices; j++)
    				{
    				if ((signal[i][j] = (int****) calloc (num_buy_channels, sizeof(int ***))) == NULL)
    					printf("couldn't allocated memory\n");
    				for (k = 0; k < num_buy_channels; k++)
    					{//printf("c");
    					if ((signal[i][j][k] = (int***) calloc (num_sell_channels, sizeof(int **))) == NULL)
    					printf("couldn't allocated memory\n");
    					for (l = 0; l < num_sell_channels; l++)
    						{//printf("d\n");
    							if ((signal[i][j][k][l] = (int**) calloc(2, sizeof(int *))) == NULL)
    								printf("couldn't allocated memory\n");
    						for (m = 0; m < 2; m++) // number of fk values
    						if	( (signal[i][j][k][l][m] = (int*) calloc(days_of_data, sizeof(int ))) == NULL)
    							printf("couldn't allocated memory\n");
    						}
    					}
    				}
    			} 
    
     
  2. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
    #2
    Open up Activity monitor and check if you are running out of physical memory and are starting to use virtual memory. That would really slow things down.
     
  3. thread starter macrumors 6502a

    Joined:
    Sep 16, 2008
    #3
    Not sure what I see when I open the Activity Monitor. I'll ask my kid when I see him.
    Thanks.
     
  4. macrumors 6502

    ElectricSheep

    Joined:
    Feb 18, 2004
    Location:
    Wilmington, DE
    #4
    Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc(). Unless you actually need to manage any of those allocations, I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.
     
  5. thread starter macrumors 6502a

    Joined:
    Sep 16, 2008
    #5
    That seems to work although I now get an error later when signal ( and other things are sent to a procedure). That's for tomorrow.
    Thanks
     
  6. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
  7. macrumors 68040

    lee1210

    Joined:
    Jan 10, 2005
    Location:
    Dallas, TX
    #7
    Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

    It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

    -Lee
     
  8. macrumors 68040

    Joined:
    Feb 2, 2008
    #8
    No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.

    ----------

    Yep nuts, the OS has to keep track of 320,000 pointers for 1 process alone for what is essentially one object.
     
  9. macrumors G5

    gnasher729

    Joined:
    Nov 25, 2005
    #9
    So the solution is:

    1. Upgrade RAM. 8 GB for number_of_indices < 50, 16 GB for number_of_indices < 70.
    2. If possible, store "short" instead of int; that gets you twice as far.
    3. Change the algorithm. Whatever the algorithm is.

    I would be quite surprised if these billions of array elements each contained actually useful data.
     
  10. ElectricSheep, Aug 11, 2013
    Last edited by a moderator: Aug 11, 2013

    macrumors 6502

    ElectricSheep

    Joined:
    Feb 18, 2004
    Location:
    Wilmington, DE
    #10
    That occurred to me after I posted, but I was rather tired. It is still possible to perform a single large allocation, and manually construct the multidimensional array.

    Realistically, it seems like this application might benefit from a relational model, especially if the data is sparse.
     
  11. thread starter macrumors 6502a

    Joined:
    Sep 16, 2008
    #11
    1. sizeof(int*****)
    yes, it helps me keep track of where I am.
    2. Yes every element in the array is filled.
    3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
    4. Time to rethink.
     
  12. macrumors 6502a

    Joined:
    May 10, 2009
    Location:
    Des Moines, WA
    #12
    What computer, OS version, hard drive free space and amount of RAM is this running on?
     
  13. macrumors G5

    gnasher729

    Joined:
    Nov 25, 2005
    #13
    I don't doubt that every array element is filled, but mostly with calculated values that don't need to be stored.

    The first three are quite irrelevant; the fourth one is "not enough".
     
  14. macrumors 6502a

    Joined:
    May 10, 2009
    Location:
    Des Moines, WA
    #14
    Bad response.

    If you had all the answers you wouldn't be here.
     
  15. macrumors 68000

    Joined:
    Mar 29, 2009
    Location:
    USA
    #15
    Computer error at ______ crashes market.
     

Share This Page