shared memory

[farmerdoug]

Once I get an address for shared memory for a structure:
mfilters = (matchedfilter*) shmat(shmid, NULL, 0)).
How to I write to the structure?
typedef struct matchedfilter{
int col;
int row;
int *fpcol;
int *fprow;
float ** filters;
} matchedfilter;

When not using shared memory, one would calloc memory, but doing that in the code that writes to the shared memory won't work.

[Cromulent]

Once I get an address for shared memory for a structure:
mfilters = (matchedfilter*) shmat(shmid, NULL, 0)).
How to I write to the structure?
typedef struct matchedfilter{
int col;
int row;
int *fpcol;
int *fprow;
float ** filters;
} matchedfilter;

When not using shared memory, one would calloc memory, but doing that in the code that writes to the shared memory won't work.

The same way as you do any other memory. The return of shmat() returns a void pointer that points to the start address of the shared memory which you allocated with shmget().

I assume that is correct having read the man pages. I have not done any work with shared memory really, but it seems fairly straight forward. Just treat the void pointer that is returned the same way as you would treat the void pointer that is returned from the calloc / malloc / realloc family of functions with the proviso that you take heed of the permission flags you set when creating the shared memory.

[lee1210]

Just don't try to free the thing.

Once you have the pointer, whenever you read from or write to that memory it will be immediately in the segment, so other programs will see the changes right away, it will get the most recent changes, etc.

One important piece of this, though, is locking. You may want to set up a semaphore or two to control read/write access to the shared memory segment so you don't smash changes from one process with that of another.

I don't really understand what is "global" about the structure you're using, such that it would need to be accessed across processes. You may want to explain your intent and there might be a different way to approach this problem.

-Lee

EDIT: Didn't really say specifically how to write... but looking at what you've got there, you don't want something like that in shared memory anyhow. You have all sorts of pointers in the structure which you will not be able to keep valid, at least not very effectively. You should only put concrete types in shared memory. I'll alter the structure a bit and give an example:

typedef struct matchedfilter{
int col;
int row;
int fpcol[20];
int fprow[20];
float filters[1000][1000];
} matchedfilter;
int myVal = 2;
mfilters = (matchedfilter*) shmat(shmid, NULL, 0)).
mfilters->fpcol[0] = mfilters->col * mfilters->row * myVal; //Shared memory is immediately changed
mfilters->filters[0][0] = mfilters->fpcol[0]/.5; //Shared memory is immediately changed

[jpyc7]

Not that disagree with lee1210's advice about the pointers, but it is possible to use pointers in shared memory.

I've used the shm_open and mmap calls in Linux, so I'm not too sure about the old-style shmat. Anyway, you can pass your desired memory location to mmap() so that all processes will be using the same addresses. (Note: this defeats address space layout randomization (aslr).) Then you can point within the shared memory space. Obviously, you cannot point outside shared memory. This requires you to write your own memory allocation/deallocation functions to work out of your shared memory.

Also, you may know this, but "shmdt" is used to free the memory that is allocated by shmat.

[lee1210]

Not that disagree with lee1210's advice about the pointers, but it is possible to use pointers in shared memory.

I've used the shm_open and mmap calls in Linux, so I'm not too sure about the old-style shmat. Anyway, you can pass your desired memory location to mmap() so that all processes will be using the same addresses. (Note: this defeats address space layout randomization (aslr).) Then you can point within the shared memory space. Obviously, you cannot point outside shared memory. This requires you to write your own memory allocation/deallocation functions to work out of your shared memory.

Also, you may know this, but "shmdt" is used to free the memory that is allocated by shmat.

I hadn't even thought of doing something like that. What we use shared memory for doesn't use these kinds of complex relationships and it seems a bit difficult to manage, but it's good to put it out there for someone to think about.

-Lee

[jpyc7]

I hadn't even thought of doing something like that. What we use shared memory for doesn't use these kinds of complex relationships and it seems a bit difficult to manage, but it's good to put it out there for someone to think about.

-Lee

I did it (using pointers inside shared memory) for a project at work. I opened a shared memory segment and divided that into a section for a root node of a radix tree and another section for an array of nodes that could be allocated/deallocated as the tree was modified. There was a third section for other memory that was associated with radix nodes, but whose size were variable multiples of a fixed size. If you're familiar with subnets in IPv4, basically I was tracking IPs in different subnets with sizes based on CIDR notation. So I knew the minimum subnet size (128) and that larger ones would be multiples of that.

So the memory allocation part was mostly free list-based using brute force and pointer arithmetic. The whole thing took less than 1000 lines of C with most of that for the radix tree (i.e. not dependent on memory allocation).


About the semaphores, my boss implemented circular queues in shared memory with one writer process and one reader process. In such an implementation, it is not necessary to have a semaphore. Of course, if you have more than one writer process it is necessary to do locking. A different type of data structure (like a tree) probably also requires locking.

[farmerdoug]

Thanks for all the replies. I will try and incorporate the suggestions.
As to want I am trying to do, it takes about 3 hours to fill the structure
typedef struct matchedfilter{
int col;
int row;
int *fpcol;
int *fprow;
float ** filters;
} matchedfilter;

with data from approximately 36000 files. I want to put it in shared memory so I don't have to load the data each time I run the program.

[lee1210]

This will work until you reboot... then you'll have to load it again. Maybe once you have all of your data laid out as desired in memory could just just serialize it to disk in a way that's much quicker to load in to memory each time vs. loading/parsing all 36K files?

It also looks like you'll have to work out a scheme like jpyc7 mentioned, where you can allocate "chunks" of your larger shared memory segment for use if you can't use fixed size arrays in your structure. I suppose you could setup new segments as necessary, but that seems like a big burden.

-Lee

[farmerdoug]

Here's the whole program

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>
#include <unistd.h>
#include <pipeline.h>


#define TRUE 1
#define FALSE 0

#define SHMSZ 1
int main()
{


struct dirent *filterfile;
DIR *ld;
struct matchedfilter *mfilters;
int shmid;
key_t key;
int filtnum, found;
FILE * fp;
char * filename;
int lambdainc, lambda;
int fcol, frow;
//ld = (DIR *) calloc (1, sizeof(DIR));
//if ((mfilters = (matchedfilter *) calloc(LENSLETS_SIZE*LENSLETS_SIZE, sizeof(matchedfilter) ) ) == NULL)
// printf("calloc failure\n");
filename = (char*) calloc(25, sizeof(char));
ld = opendir(filterdir);
//float value, acorr;
float *sums;
char * laserdir, *psfdir, *rootdatadir, *locfname;
int i, q, w, col, row, fpcol, fprow;
FILE **locations;
locations = (FILE **) calloc(NWAVES, sizeof(locations) );




/*
* We'll name our shared memory segment
* "5678".
*/
key = 5678;

/*
* Create the segment.
*/
if ((shmid = shmget(key, SHMSZ, IPC_CREAT | 0666)) < 0) {
perror("shmget");
return (0);
}

/*
* Now we attach the segment to our data space.
*/
if ((mfilters = (matchedfilter*) shmat(shmid, NULL, 0)) == NULL) {
perror("shmat");
return (0);
}


sums = (float*) calloc(NWAVES, sizeof(float));
filtnum = 0;
laserdir = (char *) calloc(100, sizeof(char));
psfdir = (char *) calloc(100, sizeof(char));
rootdatadir = (char *) calloc(100, sizeof(char));
locfname = (char *) calloc(100, sizeof(char));
strcpy(rootdatadir, "/DATA");
sprintf(laserdir, "%s/LASER", rootdatadir);
sprintf(psfdir, "%s/PSF", laserdir);


lambdainc = (LAMBDAEND - LAMBDASTART)/(23 - 1);
for(w = 0; w < NWAVES; w++)
{
lambda = LAMBDASTART + w*lambdainc;
sprintf(locfname, "%s/PPSuperLaser%dM.txt", psfdir, lambda);
if ( (locations[w] = fopen(locfname, "r")) == NULL)
printf("couldn't open %s\n", locfname);

}

while ( (filterfile = readdir(ld)) != NULL) //while you find a file in the directory
if ( strstr(filterfile->d_name, "txt") != NULL) //and the file is a txt file
{
strcpy(filename,filterdir);
strcat(filename,filterfile->d_name);
printf("%s\n", filename);
if ((fp = fopen(filename, "r")) != NULL) //open the file
{
if (fmod( (float)filtnum + 1,100.0) == 0)
printf("reading fitler set %d \n", filtnum + 1);
sscanf(filterfile->d_name, "%d-%d.txt", &col, &row); // get this file's point
for ( w = 0; w < NWAVES; w++) // for each wavelength get the fp point
{
found = FALSE;
while ( (fscanf(locations[w], "%d %d, %d %d\n", &fcol, &frow, &fpcol, &fprow) !=EOF) && (found == FALSE) )
{
if( (col == fcol) && (row == frow )) // once you find the point load the fpcol and fprow
{
mfilters[w].col = col;
mfilters[w].row = row;
found = TRUE;
mfilters[filtnum].fpcol[w] = fpcol;
mfilters[filtnum].fprow[w] = fprow;
fread(mfilters[filtnum].filters[w], 1, (12 + 3*w)*sizeof(float), fp);


//printf("\n%d %d %d \n", w, mfilters[filtnum].fpcol[w], mfilters[filtnum].fprow[w]);
// for(i = 0; i < 12 + 3*w; i++)
// printf("%d %f \n",w, mfilters[filtnum].filters[w][i]);


} // end if
} // end while
rewind(locations[w]);
} // for each wave
fclose(fp);

printf("%d %d \n", mfilters[filtnum].col, mfilters[filtnum].row);
for ( w = 0; w < NWAVES; w++)
{
printf("%d %d %d \n",w, mfilters[filtnum].fpcol[w], mfilters[filtnum].fprow[w]);
for(i = 0; i < 12 + 3*w; i++)
printf("%f, \n", mfilters[filtnum].filters[w][i]);
}
filtnum++;
} // end if you can open the file
} //end if checking that file ends .txt

closedir(ld);
free(filename);
for ( w = 0; w < NWAVES; w++) // for each wavelength get the fp point
fclose(locations[w]);
free(locations);


printf("%d \n", mfilters);

for (q= 0; q < 10; q++)
printf(" %d \n", (*(mfilters+ q*sizeof(int))));




return (0);
}


worth a shot. Got a reference?
Thanks?

It does load every thing into memory but I'm still having trouble accessing it.

[gnasher729]

Thanks for all the replies. I will try and incorporate the suggestions.
As to want I am trying to do, it takes about 3 hours to fill the structure
typedef struct matchedfilter{
int col;
int row;
int *fpcol;
int *fprow;
float ** filters;
} matchedfilter;

with data from approximately 36000 files. I want to put it in shared memory so I don't have to load the data each time I run the program.

Not sure if you know how shared memory works, but even though the two processes access the same memory, they are quite likely to use different pointers to access the same memory. Passing pointers from one process to another is therefore pointless. The shared memory could be at address 0x10000 in one process and at address 0x20000 in another process.

What you can do is create a shared memory segment big enough for all your data, then create a data structure similar to matchedfilter that doesn't contain pointers, but offsets to the start of the shared memory segment. Your code will obviously need changing.

Or quite possibly the two values "col" and "row" are enough to determine what data you need. In that case, write down exactly where everything is stored, and create the pointers independently in each process (all the pointers will of course point into the shared memory segment). Remember: Pointers created in one process don't work in another process.

[farmerdoug]

except that shmat returns the same pointer in both the program where I write and the program where I read. Furthermore, using pointers to check in the write program, I can't read. Does this match what you are saying? You are certainly right that my knowledge has holes in it. I took some code off the web and got it to work for int arrays so thought I would be able to get it to work here.

[lee1210]

As far as i can tell, you're getting a segment that's 1 byte. I also don't see anything setting up memory for fpcol, fprow, or filters.

Some other things that stand out:
you're dynamically allocating things to a fixed size. Just having a char[100] seems easier.
I don't see the definition of matchedfilter, NWAVES, etc. is there a .h file we're missing?
It seems like the size of fpcol, fprow, and filters could be determined at compile time, so you could use a fixed size structure. fpcol would be NWAVES long, fprow would be NWAVES long, and filters would be NWAVES long and 3*NWAVES+12 wide. If you can set this all up statically, using #define for NWAVES. The next issue is knowing how many matchedfilters you need, which seems to be based on the number of text files in a directory. Since i assume this can change, it makes it difficult for you to have a "fixed" segment size. This seems to really present a challenge. You could definitely count these files in advance, then just get a segment that's sizeof(matchedfilter)*numfiles, but between runs if the number of files change it would be hard to tell which were in memory already, and you might have to just scrap everything and start again anyway just to be safe.

-Lee

[farmerdoug]

Yes. There is an h file. You've seen parts of it.

typedef struct matchedfilter{
int col;
int row;
int fpcol[NWAVES];
int fprow[NWAVES];
float filters[NWAVES][12 + 3*NWAVES];
} matchedfilter;


and NWAVES is defined as well

I do know how many files I have. It's about 36000 but the machine doesn't like

#define SHMSZ sizeof(matchedfilter)*40000

if ((shmid = shmget(key, SHMSZ, IPC_CREAT | 0666)) < 0) {
perror("shmget");
return (0);
}
it fails. It also fails for #define SHMSZ sizeof(matchedfilter)

[lee1210]

you may want to pick an upper bounds on the number of files that can be handled, and just grab a segment that is that * sizeof(matchedfilter) bytes. If you have 36K files now, maybe allow up to 100K, 500K, etc. I don't know what size NWAVES is, but the size of matchedfilter will be 4*(3*NWAVES^2 + 14*NWAVES + 2) bytes. If NWAVES is something like 10, matchedfilter is only a couple of kilobytes. For 100K entries, this would be about 200MB. If it's more like 100, matchedfilter is ~125K.... so 100K of them would be over 12GB. I don't know for sure, but you may run into trouble asking for a 12GB segment of shared memory.

As for serialization, you can search around. This doesn't seem too complex, and if you don't need for it to be portable to other architectures you could just write out a file that has a count of matchedfilters in it's first 4/8 bytes, then each matchedfilter structure written out. Reading this back in should be pretty fast, but maybe not if it does end up being 12GB.

-Lee

I do know how many files I have. It's about 36000 but the machine doesn't like

#define SHMSZ sizeof(matchedfilter)*40000

if ((shmid = shmget(key, SHMSZ, IPC_CREAT | 0666)) < 0) {
perror("shmget");
return (0);
}
it fails. It also fails for #define SHMSZ sizeof(matchedfilter)

What kind of failure? That looks OK to me...
What if you put the formula directly into the shmget call instead of a #define (not discouraging #define, just trying to see what the problem is).

-Lee

EDIT: You might need to say sizeof(struct matchedfilter)

[farmerdoug]

if ((shmid = shmget(key, sizeof(matchedfilter), IPC_CREAT | 0666)) < 0) {
perror("shmget");
return (0);
}

shmget: Invalid argument

[lee1210]

if ((shmid = shmget(key, sizeof(matchedfilter), IPC_CREAT | 0666)) < 0) {
perror("shmget");
return (0);
}

shmget: Invalid argument

Hm. There could be a few different things going on. You may want to ipcrm the existing segment, since you're asking for a new one with the same key of a different size. If you still have problems print errno and you can see what the reason is for the failure.

-Lee

[farmerdoug]

NWAVES is 32 so we are talking about 70Mbytes which shouldn't be a lot.

sizeof(struct matchedfilter) didn't help

Would you send me a phone number? dbrenner@amnh.org

Almost there. Cleared the memory was able to get it to work for NWAVES = 4
Strange thing I get the right answer from my read program but not from the test print out in the write program.

[lee1210]

NWAVES is 32 so we are talking about 70Mbytes which shouldn't be a lot.

sizeof(struct matchedfilter) didn't help

Would you send me a phone number? dbrenner@amnh.org

By my calculations w/ NWAVES = 32, you'll need ~500MB to store 40K.

I'm happy to help here, but don't want to give out personal contact info, sorry.

ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x000109ca 65538 prodigy 664 7733033 5
0x000209ca 98307 prodigy 664 145001 5
0x000309ca 131076 prodigy 664 9 5
0x000409ca 163845 prodigy 664 30003 5
0x000509ca 196614 prodigy 664 680461 5


Identify the right segment, it should end with 162E. Then you can ipcrm -m <shmid> with the ID on the corresponding line. (It looks like you already figured this out)

Does it stop working if NWAVES is 5? 10? 100? You'll need to delete the segment each time to try a new value.

-Lee

[farmerdoug]

I tend to read these posts too fast. 500 MB may indeed be too large, I only have 4GB on the machine. I'll look into it some more. I understand about contact info. I didn't want to ask you to call me; then it would be on your dime. Anyway you win the prize. Get to NYC. email me for tickets to the Museum of Natural History.

[lee1210]

I'm not entirely familiar with all of this on OS X, mostly deal with it on linux, but in /etc/rc there should be a line:
kern.sysv.shmmax = XXXXXX

The value on that line is the maximum number of bytes for a shared memory segment. You should be able to go up to this amount, and if you really need you can tweak this if it's too low for your needs.

i doubt you'd ruin anything, but i'd make a copy of rc before changing it, so you can easily restore it from single user mode, etc. in case something bad happens.

-Lee

[farmerdoug]

So I was able to increase shmmax to over 500 MB and get to acquire memory. I may still have some problems they may lie elsewhere.

FYI on my MAC running 10.5.6

cd /usr/sbin
sudo sysctl -w kern.sysv.shmmax=536870912

sudo to run as a superuser and get permission.
BTW, the change did not survive rebooting.

[lee1210]

http://joseph.randomnetworks.com/archives/2004/07/06/tweaking-mac-os-x-sysctl-values/

That post has a discussion about this. The sysctl command will change things for the current session, but you'll need to change the value in /etc/rc or all shmem values in sysctl.conf for this to survive a reboot.

You could have multiple segments instead, and switch whenever your index mod a particular value equals 0. It adds some complexity, but may help you avoid system level changes.

-Lee

[farmerdoug]

So it runs or so it seems but 500 files and NWAVES= 32 requires 1 GB of memory??

http://www.pythian.com/news/245/the-mysterious-world-of-shmmax-and-shmall

You have to set shmall

[lee1210]

So it runs or so it seems but 500 files and NWAVES= 32 requires 1 GB of memory??

Hm. Instead of using my back-of-the-envelope math to estimate the usage, you should probably just print sizeof(matchedfilter), and/or that times 500 and see how many bytes you'll actually need. By my math with NWAVES at 32, matchedfilter should be about 14K, so for 500 files you should only need about 7MB. I could have botched that formula, though, so just printing in the code should definitely be more exact.

-Lee

[farmerdoug]

Sizeof(matchedfilter) gave me about 7000. When I change shmall, everything fell into place. I left it happily chugging along.
Thanks.

[farmerdoug]

Last I successfully loaded everything into memory. This morning a modified by main program to read the memory by cutting and pasting from my test read program. Naturally, it didn't work. So I printed out the address that both programs were using and out different ones. ??

[jpyc7]

Last I successfully loaded everything into memory. This morning a modified by main program to read the memory by cutting and pasting from my test read program. Naturally, it didn't work. So I printed out the address that both programs were using and out different ones. ??

If you are using using the pointer returned from shmat(), then the different addresses should not matter. You could try passing a fixed address to shmat if you really wanted. As I recall (without looking at the posts from yesterday), you took out all the pointers within the structure. Most likely you introduced a bug somewhere during the copy-and-paste.

[lee1210]

Are you looking at the address returned from shmat? This will vary from call to call. There's not a global address space from which you get a pointer to the same "place". Each process has its own memory space, and shmat will "map" the shared memory managed by the system into your processes address space.

So that, in and of itself, is not a problem. Can you try with one file, and see if you can save the structure, then try to read it back out with a separate program? Are you sure your NWAVES is the same on both so the size of the structure is the same?

-Lee

[farmerdoug]

Never assume.
I put in a print statement right after getting the address and at that point everything seemed okay.
Thanks guys. You are really a big help.