macOS Matlab and Concatenating in an order

[dukebound85]

Hey all, had a quick question

Say I have a few 3d arrays with dimensions of 240x40x8094. Let's call them x,y,z and w. How can I put them together into a matrix of 240x40x32376 called say t?

For instance, I want them to be put together in such a way as

(1,1,t1) = (1,1,x1)
(1,1,t2) = (1,1,y1)
(1,1,t3) = (1,1,z1)
(1,1,t4) = (1,1,w1)
(1,1,t5) = (1,1,x2)
(1,1,t6) = (1,1,y2)

etc, where x1,y1,z1,x2 is the time dimension in increasing order


The first two dimensions are the same for each matrix

Here is what I have tried so far with no luck. The first part is initializing the arrays. The 'cat' variables are the arrays just appended to each other ie
(1,1,tcat1) = (1,1,x1)
(1,1,tcat2) = (1,1,x2)
.
.
(1,1,tcat8095) = (1,1,y1)
etc

The t_cat matrix is what I expect. The t matrix is not

The third dimension is time and if I sort by time in ascending order, I would expect all values associated with the third index (ie the other dimensions) would be sorted along with it

t_cat = zeros(240,40,32376);
t = zeros(240,40,32376);
 
t_cat = cat(3,x,y,z,w);

t = sort(t_cat,3,'ascend');

I feel like I am close and not sure why the above is not working but all I know is that it is not

If any of you can offer some insight, that would be great

 

[crackpip]

The code you pasted generates what I would expect on a few smaller test arrays. What do you get when you run it?

Also, I assume you've got plenty of RAM (for the numbers you gave you'll be using 7.1 GB of RAM).

crackpip

 

[balamw]

Also, I assume you've got plenty of RAM (for the numbers you gave you'll be using 7.1 GB of RAM).

This is why in a case like this I might try to only sort the third dimension and use that as an index rather than creating a duplicate huge data structure.

You could use the

[B,IX] = sort(A,...)

form of sort that returns the index for this.

B

 

[dukebound85]

The code you pasted generates what I would expect on a few smaller test arrays. What do you get when you run it?

Also, I assume you've got plenty of RAM (for the numbers you gave you'll be using 7.1 GB of RAM).

crackpip

I have 8gigs and it makes my computer crawl to a stop pretty much lol

when I do it, it doesn't pass my check, which is

(1,1,x1) = (1,1,t1)

When I use my code, it seems to take the values at that index and also sorts them in ascending order. IE (1,1,x1) = 8.5 and (1,1,t1) = -27 (which is the lowest in the whole data set....which goes along with it being sorted in an ascending manner

This is why in a case like this I might try to only sort the third dimension and use that as an index rather than creating a duplicate huge data structure.

You could use the

[B,IX] = sort(A,...)

form of sort that returns the index for this.

B

I was attempting to try that. I made a 1d array with time to get the right index. I was able to sort that index in the desired manner. Then to apply that index to the t matrix, I am trying this

[time,index] = sort(time,1,'ascend');

t = t_cat,:,index);

See any flaws?

Thanks for the input so far

 

[crackpip]

I have 8gigs and it makes my computer crawl to a stop pretty much lol

when I do it, it doesn't pass my check, which is

(1,1,x1) = (1,1,t1)

When I use my code, it seems to take the values at that index and also sorts them in ascending order. IE (1,1,x1) = 8.5 and (1,1,t1) = -27 (which is the lowest in the whole data set....which goes along with it being sorted in an ascending manner

Are the times the same for each array? Because then the sort function would use a value from the 1st dim to determine the order. Do you always expect the order to be X1,Y1,Z1,W1... when you sort by time? If so you might be better off just doing a loop (I.e for n=1:8094 t,:,(n-1)*4+1)=X,:,n); t,:,(n-1)*4+2)=Y,:,n); etc.).

Furthermore by doing a loop, you only need one giant array and if your reading the smaller arrays, you could do one at a time and reuse the smaller array variable, too. Reducing your memory foot print would probably speed it up quite a bit.

Crackpip

 

[dukebound85]

Are the times the same for each array? Because then the sort function would use a value from the 1st dim to determine the order. Do you always expect the order to be X1,Y1,Z1,W1... when you sort by time? If so you might be better off just doing a loop (I.e for n=1:8094 t,:,(n-1)*4+1)=X,:,n); t,:,(n-1)*4+2)=Y,:,n); etc.).

Furthermore by doing a loop, you only need one giant array and if your reading the smaller arrays, you could do one at a time and reuse the smaller array variable, too. Reducing your memory foot print would probably speed it up quite a bit.

Crackpip

No, the times are not the same.The length of the time dimension is the same for each however. For instance,
In X, the times are 0,24,48,72,96, etc
In Y, the times are 6,30,54,78,102
In Z, the times are 12,36,60,84,108
In W, the times are 18,42,66,90,114

Essentially the times are staggered and I want the final result to be
X1,Y1,Z1,W1,X2,Y2,Z2,W2,etc which is 0,6,12,18,24,30,etc and have the entire 3d array be sorted respectively in that time dimension (as there are properties at each time that I want to preserve)

I may try a loop but have been worried as I have always thought loops were much more time consuming than keeping things in vectors and merely manipulating vectors directly. Thanks for the suggestion, I will be sure to look into it

Anything to cut down on my memory usage would be great lol

 

[crackpip]

I may try a loop but have been worried as I have always thought loops were much more time consuming than keeping things in vectors and merely manipulating vectors directly. Thanks for the suggestion, I will be sure to look into it

Anything to cut down on my memory usage would be great lol

In general using vectors are more efficient. Notice that in the loop I suggested the index on the right is the one with the loop variable. MATLAB stores data like fortran in column major format, so X(1,1,1) and X(2,1,1) are adjacent in memory and therefore X,:,n) will be a bunch of values next to each other in memory and will get a speed boost.

Also no matter what, if your program has to resort to virtual memory paging it will be incredibly slow. Any benefit you would get from doing the whole array operations will be dwarfed by MATLAB having to page in/out.

Crackpip

 

[dukebound85]

Thanks for the help. I did manage to figure out the vector way to do it and with your help, the loop way. I will compare speeds for the heck of it

As per the vector way, all I had to do was

t = t_cat,:,index);

where index was given by
[time,index] = sort(time_cat,'ascend');

This resulted in the desired array