Go Back   MacRumors Forums > Apple Systems and Services > Programming > Mac Programming

Reply
 
Thread Tools Search this Thread Display Modes
Old Aug 10, 2013, 07:31 PM   #1
farmerdoug
macrumors 6502a
 
Join Date: Sep 2008
speed of memory allocation

Once number_of_inidices gets over 40, things get very slow.
number_of_buy_channels = number_of_sell_channels ~ 10
days_of_data = 3400

Suggestions?
thanks

Code:
signal = (int******) calloc (number_of_indices, sizeof(int*****));
		for (i = 0; i < number_of_indices; i++)
			{
			if( (signal[i] = (int*****) calloc (number_of_indices, sizeof(int ****))) == NULL)
				printf("couldn't allocated memory\n");
			for (j = 0; j < number_of_indices; j++)
				{
				if ((signal[i][j] = (int****) calloc (num_buy_channels, sizeof(int ***))) == NULL)
					printf("couldn't allocated memory\n");
				for (k = 0; k < num_buy_channels; k++)
					{//printf("c");
					if ((signal[i][j][k] = (int***) calloc (num_sell_channels, sizeof(int **))) == NULL)
					printf("couldn't allocated memory\n");
					for (l = 0; l < num_sell_channels; l++)
						{//printf("d\n");
							if ((signal[i][j][k][l] = (int**) calloc(2, sizeof(int *))) == NULL)
								printf("couldn't allocated memory\n");
						for (m = 0; m < 2; m++) // number of fk values
						if	( (signal[i][j][k][l][m] = (int*) calloc(days_of_data, sizeof(int ))) == NULL)
							printf("couldn't allocated memory\n");
						}
					}
				}
			}
farmerdoug is offline   0 Reply With Quote
Old Aug 10, 2013, 07:35 PM   #2
robvas
macrumors 68000
 
Join Date: Mar 2009
Location: USA
Open up Activity monitor and check if you are running out of physical memory and are starting to use virtual memory. That would really slow things down.
robvas is offline   0 Reply With Quote
Old Aug 10, 2013, 07:46 PM   #3
farmerdoug
Thread Starter
macrumors 6502a
 
Join Date: Sep 2008
Not sure what I see when I open the Activity Monitor. I'll ask my kid when I see him.
Thanks.
farmerdoug is offline   0 Reply With Quote
Old Aug 10, 2013, 07:54 PM   #4
ElectricSheep
macrumors 6502
 
Join Date: Feb 2004
Location: Wilmington, DE
Send a message via AIM to ElectricSheep
Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc(). Unless you actually need to manage any of those allocations, I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.
__________________
15'' MBP (early 2011) | i7 3770k Hackintosh | i7 Mac Mini (late 2012) | iPhone 5 | iPad 3 (2012) | iPad mini | MacOS X 10.9.2
ElectricSheep is offline   0 Reply With Quote
Old Aug 10, 2013, 07:59 PM   #5
farmerdoug
Thread Starter
macrumors 6502a
 
Join Date: Sep 2008
That seems to work although I now get an error later when signal ( and other things are sent to a procedure). That's for tomorrow.
Thanks
farmerdoug is offline   0 Reply With Quote
Old Aug 10, 2013, 10:35 PM   #6
robvas
macrumors 68000
 
Join Date: Mar 2009
Location: USA
Using Activity Monitor to read System Memory and determine how much RAM is being used

sizeof(int*****)

Seriously?
robvas is offline   0 Reply With Quote
Old Aug 10, 2013, 11:01 PM   #7
lee1210
macrumors 68040
 
lee1210's Avatar
 
Join Date: Jan 2005
Location: Dallas, TX
Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

-Lee
lee1210 is offline   0 Reply With Quote
Old Aug 11, 2013, 09:26 AM   #8
subsonix
macrumors 68040
 
Join Date: Feb 2008
Quote:
Originally Posted by ElectricSheep View Post
I would suggest just allocating all of the space you need in one large chunk; you can still use array subscripting to access each dimension.
No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.

----------

Quote:
Originally Posted by ElectricSheep View Post
Just doing some math, you are performing 40*40*10*10*2 allocations, which is 320,000 calls to calloc().
Yep nuts, the OS has to keep track of 320,000 pointers for 1 process alone for what is essentially one object.
subsonix is offline   0 Reply With Quote
Old Aug 11, 2013, 11:42 AM   #9
gnasher729
macrumors G5
 
gnasher729's Avatar
 
Join Date: Nov 2005
Quote:
Originally Posted by lee1210 View Post
Just the inner-most loop will run the 40*40*10*10*2 times, each time allocating 3400*4 bytes. This is about 4.05GB of ram. Each drop of number of indices lowers this by about 200MB for a while. This omits the overhead of all the pointer overhead, but that's not too significant.

It seems obscene to keep this all in memory at once. Obviously we have no idea what you're doing other than stock market analysis, but surely the data could be partitioned or stored in a database to help alleviate these shenanigans.

-Lee
So the solution is:

1. Upgrade RAM. 8 GB for number_of_indices < 50, 16 GB for number_of_indices < 70.
2. If possible, store "short" instead of int; that gets you twice as far.
3. Change the algorithm. Whatever the algorithm is.

I would be quite surprised if these billions of array elements each contained actually useful data.
gnasher729 is offline   0 Reply With Quote
Old Aug 11, 2013, 12:11 PM   #10
ElectricSheep
macrumors 6502
 
Join Date: Feb 2004
Location: Wilmington, DE
Send a message via AIM to ElectricSheep
Quote:
Originally Posted by subsonix View Post
No he can't because a pointer has no idea how large memory area it point's to, or how large each dimension is, therefor it has no ability to add offsets of the right size. Using structs is probably an alternative to the dimensional array.
That occurred to me after I posted, but I was rather tired. It is still possible to perform a single large allocation, and manually construct the multidimensional array.

Realistically, it seems like this application might benefit from a relational model, especially if the data is sparse.
__________________
15'' MBP (early 2011) | i7 3770k Hackintosh | i7 Mac Mini (late 2012) | iPhone 5 | iPad 3 (2012) | iPad mini | MacOS X 10.9.2

Last edited by maflynn; Aug 11, 2013 at 03:05 PM. Reason: Fixed quote tag
ElectricSheep is offline   0 Reply With Quote
Old Aug 11, 2013, 06:35 PM   #11
farmerdoug
Thread Starter
macrumors 6502a
 
Join Date: Sep 2008
1. sizeof(int*****)
yes, it helps me keep track of where I am.
2. Yes every element in the array is filled.
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
4. Time to rethink.
farmerdoug is offline   0 Reply With Quote
Old Aug 11, 2013, 07:59 PM   #12
lloyddean
macrumors 6502a
 
Join Date: May 2009
Location: Des Moines, WA
What computer, OS version, hard drive free space and amount of RAM is this running on?
lloyddean is offline   0 Reply With Quote
Old Aug 12, 2013, 02:47 AM   #13
gnasher729
macrumors G5
 
gnasher729's Avatar
 
Join Date: Nov 2005
Quote:
Originally Posted by farmerdoug View Post
1. sizeof(int*****)
yes, it helps me keep track of where I am.
2. Yes every element in the array is filled.
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
4. Time to rethink.
I don't doubt that every array element is filled, but mostly with calculated values that don't need to be stored.

Quote:
Originally Posted by lloyddean View Post
What computer, OS version, hard drive free space and amount of RAM is this running on?
The first three are quite irrelevant; the fourth one is "not enough".
gnasher729 is offline   0 Reply With Quote
Old Aug 12, 2013, 10:42 AM   #14
lloyddean
macrumors 6502a
 
Join Date: May 2009
Location: Des Moines, WA
Bad response.

If you had all the answers you wouldn't be here.
lloyddean is offline   0 Reply With Quote
Old Aug 12, 2013, 11:00 AM   #15
robvas
macrumors 68000
 
Join Date: Mar 2009
Location: USA
Quote:
Originally Posted by farmerdoug View Post
3. This code has worked in the past for me. It is for the stock market. The way it is written allows me to do all my preprocessing before getting into actually choosing stocks and more importantly it makes for a procedure that can be ported to other stock picking code. It it no longer works because it appears I am push the machines limits.
Computer error at ______ crashes market.
robvas is offline   0 Reply With Quote

Reply
MacRumors Forums > Apple Systems and Services > Programming > Mac Programming

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Similar Threads
thread Thread Starter Forum Replies Last Post
GPU memory allocation in After Effects adamneer OS X Mavericks (10.9) 3 Oct 26, 2013 05:11 PM
Internet Speed - Do I Need More Memory? cgm707 MacBook Pro 12 Jun 23, 2013 06:28 AM
Memory speed question Will G. Mac Pro 1 Feb 9, 2013 07:47 PM
Memory allocation michial MacBook Pro 3 Nov 28, 2012 10:06 AM
More Bootcamp Issues with 2012 MBA: HD4000 Memory Allocation? goinskiing MacBook Air 38 Nov 1, 2012 09:48 AM

Forum Jump

All times are GMT -5. The time now is 06:35 AM.

Mac Rumors | Mac | iPhone | iPhone Game Reviews | iPhone Apps

Mobile Version | Fixed | Fluid | Fluid HD
Copyright 2002-2013, MacRumors.com, LLC