macOS strtok problem?

farmerdoug · Mar 4, 2013

The routine below is called twice for two separate lists. The first time I call it, it reads and loads data from 100 lines in 5 files.
The second time I call it, it only reads 6 lines from a few files before messing up on reading the date and crashing. When I print out the results of fgets, every line is properly printed.
ideas?

Code:

void  load_data(char *dir_path, char* list, float **array1, float **array2, char **date_list)
	{
		FILE *FILELIST, *SEC;
		char *name, *full_path, *line_of_data, *close_s, *open_s, **symbol_list;
		int  day, i, symbol_number = 0; 
		float close = 0, open = 0;
		
		close_s = (char *) calloc (30, sizeof(char));
		open_s = (char *) calloc (30, sizeof(char));
		name = (char *) calloc (100, sizeof(char));
		full_path = (char *) calloc (150, sizeof(char));
		line_of_data = (char *) calloc (150, sizeof(char));

		symbol_list = (char**) calloc(number_of_indices, sizeof(char*));
		for (i = 0; i <number_of_indices; i++)
			symbol_list[i] = (char *)calloc(10, sizeof(char));
		if( (FILELIST = fopen(list,"r")) == NULL)
		{
		printf( "couldn't open  file list\n");
		}


	while (fscanf(FILELIST, "%s\n", name) != EOF  && symbol_number < number_of_indices)
	{
		
		
		{
		strcpy(full_path, dir_path); strcat(full_path, name);
		if ( (SEC = fopen(full_path, "r")) == NULL)
			printf( "couldn't open index file %s\n", full_path);
			else
			printf( "opened index file %s\n", full_path);
		day = 0;
		while ( (fgets(line_of_data, 150, SEC))  && day < back_test_period)
		{	
		//	printf("%s\n", line_of_data);
			close_s = strtok (line_of_data,","); //symbol
			printf("%s\n",close_s);
			if ( day == 0)
				strcpy(symbol_list[symbol_number],close_s);
			
			close_s = strtok (NULL, ",");
			printf("%s\n",close_s);
			close_s = strtok (NULL, ",");// date
			printf("%s\n",close_s);
		//	printf("%s %s\n",close_s,date_list[day]);
			if (strcmp(close_s,date_list[day]) == 0)
			{
			
			if (strcmp(close_s, "<TICKER>") != 0)
				{
				close_s = strtok (NULL, ",");  ///00000
				printf("%s\n",close_s);
				open_s = strtok (NULL, ","); 
				printf("%s\n",open_s);
				for (i = 0; i < 3; i++)
					{
					close_s = strtok (NULL, ",");
					printf("%s\n",close_s);
					}
				}
			close = (float)atof(close_s);
			open = (float)atof(open_s);
		//	printf("%s %f\n", close_s, close);
			array1[symbol_number][day] =open;
			array2[symbol_number][day] =close;
			
			
			}
	//		printf("%d\n", day);
			day++;
		}
		fclose(SEC);
		}
		symbol_number++;
		
	}

	fclose(FILELIST);
	return ;
	
	}

gnasher729 · Mar 4, 2013

strtok is not reentrant - if two functions call strtok at the same time, it will crash. Google for "man strtok_r" and use that; the change is quite trivial.

farmerdoug · Mar 4, 2013

Unfortunately the C code I am using doesn't recognize strtok_r but the code is not reentrant anyway.

Code:

printf("loading stock data\n");	
	strcpy( dir, stock_file_directory);
	strcpy( list, stock_file_list);
	date_list = load_date (dir, list);
	load_data(dir, list, open_array,close_array, date_list);
	
	printf("loading index data\n");	
	strcpy( dir, index_file_directory);
	strcpy( list, index_file_list);
	load_data(dir , list,open_array, index_array, date_list);

Senor Cuete · Mar 4, 2013

calloc not freed

This won't solve your problem but I don't see where you release the memory you allocated with the calloc() calls. Don't you need to free() these pointers before returning from your function to avoid a memory leak?

farmerdoug · Mar 4, 2013

You're right. It didn't solve anything. But now the code is a little cleaner.

ghellquist · Mar 4, 2013

Try swapping reading index file and stock file (in the calling program). If it breaks, there probably is an error in the content of the index file. If not, the error is either in the module or in how you use the arrays. I would be suspicious about the following line:

array1[symbol_number][day] =open;

because you pass the open_array twice, it just might update the open statement on two different places depending on the exact file ordering.

//gunnar

gnasher729 · Mar 4, 2013

farmerdoug said:
Unfortunately the C code I am using doesn't recognize strtok_r but the code is not reentrant anyway.

The problem is not in your code, it is in strtok. If your code calls strtok, and some totally unrelated call in another thread calls strtok, things go wrong.

farmerdoug · Mar 4, 2013

I copied the code into Xcode and changed to strtok_r. Code still fails after reading and parsing six lines. Code already successfully parse several lines of code when this and another routine was previously called.

farmerdoug · Mar 4, 2013

The pilot found his error. Thanks guys.

Senor Cuete · Mar 6, 2013

Questions

Was the error strtok()?

My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?

In a real world application if you read a file line by line from a disk it will go slowly particularly if it's a big file because each command to read or write is a transaction and will take 10 or 15 milliseconds. Your drive can send you a big burst of continuous data rapidly if it's only one transaction.

You might want to read the whole file into memory by doing something like this:

Code:

if((fseek(inputFile, 0, SEEK_END)) != 0)//set to file End
	bailout();
if(fgetpos(inputFile, &fileSize) != 0) //filePosition = file size
	bailout();
if((fseek(inputFile, 0, SEEK_SET)) != 0)//reset to file start for reading
	bailout();
if((fileBuffer = malloc(fileSize + 1)) == NULL) //+1 for  NULL termination
	bailout();
if((bytesRead = fread(fileBuffer, sizeof(char), fileSize, inputFile)) != fileSize)  //read whole file into memory
	bailout();
fclose(inputFile);

And then parse the data in memory.

subsonix · Mar 6, 2013

Senor Cuete said:
Was the error strtok()?

My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?

strtok_r is a posix function. Since posix added a threading api, it makes sense that they accompanied it with a thread safe strtok. Standard C does not do threads, (up until C11 that is).

A more convenient alternative to strtok is strsep imo, it's also thread safe.

If your program is single threaded then strtok is fine, the potential problem is if several threads use strtok at the same time since strtok use a static variable to keep track of it's progress.

chown33 · Mar 6, 2013

Senor Cuete said:
My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?

Read the man page:

Code:

man strtok_r

Or look it up in Xcode's builtin docs.

Or google it.

In a real world application if you read a file line by line from a disk it will go slowly particularly if it's a big file because each command to read or write is a transaction and will take 10 or 15 milliseconds. Your drive can send you a big burst of continuous data rapidly if it's only one transaction.

There's no speed difference for small vs. large files. If you think otherwise, please provide some evidence or explanation.

You should also look at how buffering works in stdio (Standard I/O), because it has a significant affect on how fast data is read or written. You can run tests with buffered vs. unbuffered to see and measure the difference. Hint: It may not be as slow as you think.

Reading a file entirely into memory isn't always a viable approach. Even when it's viable, it may be unnecessary. The OP wasn't having a problem with speed, but with correctness. There's no point in trying to solve speed problems until all the correctness problems have been solved first.

Search

Search

macOS strtok problem?

farmerdoug

macrumors 6502a

gnasher729

Suspended

farmerdoug

macrumors 6502a

Senor Cuete

macrumors 6502

farmerdoug

macrumors 6502a

ghellquist

macrumors regular

gnasher729

Suspended

farmerdoug

macrumors 6502a

farmerdoug

macrumors 6502a

Senor Cuete

macrumors 6502

subsonix

macrumors 68040

chown33

Moderator

Our Staff