Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

macOS speed of memory allocation

F

farmerdoug

macrumors 6502a
Original poster
Sep 16, 2008
541
0
Once number_of_inidices gets over 40, things get very slow.
number_of_buy_channels = number_of_sell_channels ~ 10
days_of_data = 3400

Suggestions?
thanks

Code: 
signal = (int******) calloc (number_of_indices, sizeof(int*****));
		for (i = 0; i < number_of_indices; i++)
			{
			if( (signal[i] = (int*****) calloc (number_of_indices, sizeof(int ****))) == NULL)
				printf("couldn't allocated memory\n");
			for (j = 0; j < number_of_indices; j++)
				{
				if ((signal[i][j] = (int****) calloc (num_buy_channels, sizeof(int ***))) == NULL)
					printf("couldn't allocated memory\n");
				for (k = 0; k < num_buy_channels; k++)
					{//printf("c");
					if ((signal[i][j][k] = (int***) calloc (num_sell_channels, sizeof(int **))) == NULL)
					printf("couldn't allocated memory\n");
					for (l = 0; l < num_sell_channels; l++)
						{//printf("d\n");
							if ((signal[i][j][k][l] = (int**) calloc(2, sizeof(int *))) == NULL)
								printf("couldn't allocated memory\n");
						for (m = 0; m < 2; m++) // number of fk values
						if	( (signal[i][j][k][l][m] = (int*) calloc(days_of_data, sizeof(int ))) == NULL)
							printf("couldn't allocated memory\n");
						}
					}
				}
			}
 
robvas

robvas

macrumors 68040
Mar 29, 2009
3,240
629
USA
Open up Activity monitor and check if you are running out of physical memory and are starting to use virtual memory. That would really slow things down.
 
F

farmerdoug

macrumors 6502a
Original poster
Sep 16, 2008
541
0
Not sure what I see when I open the Activity Monitor. I'll ask my kid when I see him.
Thanks.
 
ElectricSheep

ElectricSheep

macrumors 6502
Feb 18, 2004
498
4
Wilmington, DE
Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc(). Unless you actually need to manage any of those allocations, I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.
 
F

farmerdoug

macrumors 6502a
Original poster
Sep 16, 2008
541
0
That seems to work although I now get an error later when signal ( and other things are sent to a procedure). That's for tomorrow.
Thanks
 
lee1210

lee1210

macrumors 68040
Jan 10, 2005
3,182
3
Dallas, TX
Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

-Lee
 
S

subsonix

macrumors 68040
Feb 2, 2008
3,551
79
ElectricSheep said:
I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.
Click to expand...

No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.

----------
ElectricSheep said:
Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc().
Click to expand...

Yep nuts, the OS has to keep track of 320,000 pointers for 1 process alone for what is essentially one object.
 
gnasher729

gnasher729

Suspended
Nov 25, 2005
17,980
5,565
lee1210 said:
Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

-Lee
Click to expand...

So the solution is:

1. Upgrade RAM. 8 GB for number_of_indices < 50, 16 GB for number_of_indices < 70.
2. If possible, store "short" instead of int; that gets you twice as far.
3. Change the algorithm. Whatever the algorithm is.

I would be quite surprised if these billions of array elements each contained actually useful data.
 
ElectricSheep

ElectricSheep

macrumors 6502
Feb 18, 2004
498
4
Wilmington, DE
subsonix said:
No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.
Click to expand...

That occurred to me after I posted, but I was rather tired. It is still possible to perform a single large allocation, and manually construct the multidimensional array.

Realistically, it seems like this application might benefit from a relational model, especially if the data is sparse.
 
Last edited by a moderator:
F

farmerdoug

macrumors 6502a
Original poster
Sep 16, 2008
541
0
1. sizeof(int*****)
yes, it helps me keep track of where I am.
2. Yes every element in the array is filled.
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
4. Time to rethink.
 
gnasher729

gnasher729

Suspended
Nov 25, 2005
17,980
5,565
farmerdoug said:
1. sizeof(int*****)
yes, it helps me keep track of where I am.
2. Yes every element in the array is filled.
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
4. Time to rethink.
Click to expand...

I don't doubt that every array element is filled, but mostly with calculated values that don't need to be stored.

lloyddean said:
What computer, OS version, hard drive free space and amount of RAM is this running on?
Click to expand...

The first three are quite irrelevant; the fourth one is "not enough".
 
robvas

robvas

macrumors 68040
Mar 29, 2009
3,240
629
USA
farmerdoug said:
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
Click to expand...
Computer error at ______ crashes market.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom