strtok problem?

Discussion in 'Mac Programming' started by farmerdoug, Mar 4, 2013.

  1. macrumors 6502a

    The routine below is called twice for two separate lists. The first time I call it, it reads and loads data from 100 lines in 5 files.
    The second time I call it, it only reads 6 lines from a few files before messing up on reading the date and crashing. When I print out the results of fgets, every line is properly printed.

    void  load_data(char *dir_path, char* list, float **array1, float **array2, char **date_list)
    		char *name, *full_path, *line_of_data, *close_s, *open_s, **symbol_list;
    		int  day, i, symbol_number = 0; 
    		float close = 0, open = 0;
    		close_s = (char *) calloc (30, sizeof(char));
    		open_s = (char *) calloc (30, sizeof(char));
    		name = (char *) calloc (100, sizeof(char));
    		full_path = (char *) calloc (150, sizeof(char));
    		line_of_data = (char *) calloc (150, sizeof(char));
    		symbol_list = (char**) calloc(number_of_indices, sizeof(char*));
    		for (i = 0; i <number_of_indices; i++)
    			symbol_list[i] = (char *)calloc(10, sizeof(char));
    		if( (FILELIST = fopen(list,"r")) == NULL)
    		printf( "couldn't open  file list\n");
    	while (fscanf(FILELIST, "%s\n", name) != EOF  && symbol_number < number_of_indices)
    		strcpy(full_path, dir_path); strcat(full_path, name);
    		if ( (SEC = fopen(full_path, "r")) == NULL)
    			printf( "couldn't open index file %s\n", full_path);
    			printf( "opened index file %s\n", full_path);
    		day = 0;
    		while ( (fgets(line_of_data, 150, SEC))  && day < back_test_period)
    		//	printf("%s\n", line_of_data);
    			close_s = strtok (line_of_data,","); //symbol
    			if ( day == 0)
    			close_s = strtok (NULL, ",");
    			close_s = strtok (NULL, ",");// date
    		//	printf("%s %s\n",close_s,date_list[day]);
    			if (strcmp(close_s,date_list[day]) == 0)
    			if (strcmp(close_s, "<TICKER>") != 0)
    				close_s = strtok (NULL, ",");  ///00000
    				open_s = strtok (NULL, ","); 
    				for (i = 0; i < 3; i++)
    					close_s = strtok (NULL, ",");
    			close = (float)atof(close_s);
    			open = (float)atof(open_s);
    		//	printf("%s %f\n", close_s, close);
    			array1[symbol_number][day] =open;
    			array2[symbol_number][day] =close;
    	//		printf("%d\n", day);
    	return ;
  2. macrumors G5


    strtok is not reentrant - if two functions call strtok at the same time, it will crash. Google for "man strtok_r" and use that; the change is quite trivial.
  3. macrumors 6502a

    Unfortunately the C code I am using doesn't recognize strtok_r but the code is not reentrant anyway.
    printf("loading stock data\n");	
    	strcpy( dir, stock_file_directory);
    	strcpy( list, stock_file_list);
    	date_list = load_date (dir, list);
    	load_data(dir, list, open_array,close_array, date_list);
    	printf("loading index data\n");	
    	strcpy( dir, index_file_directory);
    	strcpy( list, index_file_list);
    	load_data(dir , list,open_array, index_array, date_list);
  4. macrumors regular

    calloc not freed

    This won't solve your problem but I don't see where you release the memory you allocated with the calloc() calls. Don't you need to free() these pointers before returning from your function to avoid a memory leak?
  5. macrumors 6502a

    You're right. It didn't solve anything. But now the code is a little cleaner.
  6. ghellquist, Mar 4, 2013
    Last edited: Mar 4, 2013

    macrumors regular

    Try swapping reading index file and stock file (in the calling program). If it breaks, there probably is an error in the content of the index file. If not, the error is either in the module or in how you use the arrays. I would be suspicious about the following line:

    array1[symbol_number][day] =open;

    because you pass the open_array twice, it just might update the open statement on two different places depending on the exact file ordering.

  7. macrumors G5


    The problem is not in your code, it is in strtok. If your code calls strtok, and some totally unrelated call in another thread calls strtok, things go wrong.
  8. macrumors 6502a

    I copied the code into Xcode and changed to strtok_r. Code still fails after reading and parsing six lines. Code already successfully parse several lines of code when this and another routine was previously called.
  9. macrumors 6502a

    The pilot found his error. Thanks guys.
  10. macrumors regular


    Was the error strtok()?

    My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?

    In a real world application if you read a file line by line from a disk it will go slowly particularly if it's a big file because each command to read or write is a transaction and will take 10 or 15 milliseconds. Your drive can send you a big burst of continuous data rapidly if it's only one transaction.

    You might want to read the whole file into memory by doing something like this:

    if((fseek(inputFile, 0, SEEK_END)) != 0)//set to file End
    if(fgetpos(inputFile, &fileSize) != 0) //filePosition = file size
    if((fseek(inputFile, 0, SEEK_SET)) != 0)//reset to file start for reading
    if((fileBuffer = malloc(fileSize + 1)) == NULL) //+1 for  NULL termination
    if((bytesRead = fread(fileBuffer, sizeof(char), fileSize, inputFile)) != fileSize)  //read whole file into memory
    And then parse the data in memory.
  11. macrumors 68040

    strtok_r is a posix function. Since posix added a threading api, it makes sense that they accompanied it with a thread safe strtok. Standard C does not do threads, (up until C11 that is).

    A more convenient alternative to strtok is strsep imo, it's also thread safe.

    If your program is single threaded then strtok is fine, the potential problem is if several threads use strtok at the same time since strtok use a static variable to keep track of it's progress.
  12. macrumors 603

    Read the man page:
    man strtok_r
    Or look it up in Xcode's builtin docs.

    Or google it.

    There's no speed difference for small vs. large files. If you think otherwise, please provide some evidence or explanation.

    You should also look at how buffering works in stdio (Standard I/O), because it has a significant affect on how fast data is read or written. You can run tests with buffered vs. unbuffered to see and measure the difference. Hint: It may not be as slow as you think.

    Reading a file entirely into memory isn't always a viable approach. Even when it's viable, it may be unnecessary. The OP wasn't having a problem with speed, but with correctness. There's no point in trying to solve speed problems until all the correctness problems have been solved first.

Share This Page