PDA

View Full Version : strtok problem?




farmerdoug
Mar 4, 2013, 06:23 AM
The routine below is called twice for two separate lists. The first time I call it, it reads and loads data from 100 lines in 5 files.
The second time I call it, it only reads 6 lines from a few files before messing up on reading the date and crashing. When I print out the results of fgets, every line is properly printed.
ideas?


void load_data(char *dir_path, char* list, float **array1, float **array2, char **date_list)
{
FILE *FILELIST, *SEC;
char *name, *full_path, *line_of_data, *close_s, *open_s, **symbol_list;
int day, i, symbol_number = 0;
float close = 0, open = 0;

close_s = (char *) calloc (30, sizeof(char));
open_s = (char *) calloc (30, sizeof(char));
name = (char *) calloc (100, sizeof(char));
full_path = (char *) calloc (150, sizeof(char));
line_of_data = (char *) calloc (150, sizeof(char));

symbol_list = (char**) calloc(number_of_indices, sizeof(char*));
for (i = 0; i <number_of_indices; i++)
symbol_list[i] = (char *)calloc(10, sizeof(char));
if( (FILELIST = fopen(list,"r")) == NULL)
{
printf( "couldn't open file list\n");
}


while (fscanf(FILELIST, "%s\n", name) != EOF && symbol_number < number_of_indices)
{


{
strcpy(full_path, dir_path); strcat(full_path, name);
if ( (SEC = fopen(full_path, "r")) == NULL)
printf( "couldn't open index file %s\n", full_path);
else
printf( "opened index file %s\n", full_path);
day = 0;
while ( (fgets(line_of_data, 150, SEC)) && day < back_test_period)
{
// printf("%s\n", line_of_data);
close_s = strtok (line_of_data,","); //symbol
printf("%s\n",close_s);
if ( day == 0)
strcpy(symbol_list[symbol_number],close_s);

close_s = strtok (NULL, ",");
printf("%s\n",close_s);
close_s = strtok (NULL, ",");// date
printf("%s\n",close_s);
// printf("%s %s\n",close_s,date_list[day]);
if (strcmp(close_s,date_list[day]) == 0)
{

if (strcmp(close_s, "<TICKER>") != 0)
{
close_s = strtok (NULL, ","); ///00000
printf("%s\n",close_s);
open_s = strtok (NULL, ",");
printf("%s\n",open_s);
for (i = 0; i < 3; i++)
{
close_s = strtok (NULL, ",");
printf("%s\n",close_s);
}
}
close = (float)atof(close_s);
open = (float)atof(open_s);
// printf("%s %f\n", close_s, close);
array1[symbol_number][day] =open;
array2[symbol_number][day] =close;


}
// printf("%d\n", day);
day++;
}
fclose(SEC);
}
symbol_number++;

}

fclose(FILELIST);
return ;

}



gnasher729
Mar 4, 2013, 07:05 AM
strtok is not reentrant - if two functions call strtok at the same time, it will crash. Google for "man strtok_r" and use that; the change is quite trivial.

farmerdoug
Mar 4, 2013, 07:29 AM
Unfortunately the C code I am using doesn't recognize strtok_r but the code is not reentrant anyway.

printf("loading stock data\n");
strcpy( dir, stock_file_directory);
strcpy( list, stock_file_list);
date_list = load_date (dir, list);
load_data(dir, list, open_array,close_array, date_list);

printf("loading index data\n");
strcpy( dir, index_file_directory);
strcpy( list, index_file_list);
load_data(dir , list,open_array, index_array, date_list);

Senor Cuete
Mar 4, 2013, 08:25 AM
This won't solve your problem but I don't see where you release the memory you allocated with the calloc() calls. Don't you need to free() these pointers before returning from your function to avoid a memory leak?

farmerdoug
Mar 4, 2013, 08:42 AM
You're right. It didn't solve anything. But now the code is a little cleaner.

ghellquist
Mar 4, 2013, 08:56 AM
Try swapping reading index file and stock file (in the calling program). If it breaks, there probably is an error in the content of the index file. If not, the error is either in the module or in how you use the arrays. I would be suspicious about the following line:

array1[symbol_number][day] =open;

because you pass the open_array twice, it just might update the open statement on two different places depending on the exact file ordering.


//gunnar

gnasher729
Mar 4, 2013, 11:28 AM
Unfortunately the C code I am using doesn't recognize strtok_r but the code is not reentrant anyway.

The problem is not in your code, it is in strtok. If your code calls strtok, and some totally unrelated call in another thread calls strtok, things go wrong.

farmerdoug
Mar 4, 2013, 01:32 PM
I copied the code into Xcode and changed to strtok_r. Code still fails after reading and parsing six lines. Code already successfully parse several lines of code when this and another routine was previously called.

farmerdoug
Mar 4, 2013, 01:50 PM
The pilot found his error. Thanks guys.

Senor Cuete
Mar 6, 2013, 08:16 AM
Was the error strtok()?

My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?

In a real world application if you read a file line by line from a disk it will go slowly particularly if it's a big file because each command to read or write is a transaction and will take 10 or 15 milliseconds. Your drive can send you a big burst of continuous data rapidly if it's only one transaction.

You might want to read the whole file into memory by doing something like this:


if((fseek(inputFile, 0, SEEK_END)) != 0)//set to file End
bailout();
if(fgetpos(inputFile, &fileSize) != 0) //filePosition = file size
bailout();
if((fseek(inputFile, 0, SEEK_SET)) != 0)//reset to file start for reading
bailout();
if((fileBuffer = malloc(fileSize + 1)) == NULL) //+1 for NULL termination
bailout();
if((bytesRead = fread(fileBuffer, sizeof(char), fileSize, inputFile)) != fileSize) //read whole file into memory
bailout();
fclose(inputFile);


And then parse the data in memory.

subsonix
Mar 6, 2013, 08:40 AM
Was the error strtok()?

My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?


strtok_r is a posix function. Since posix added a threading api, it makes sense that they accompanied it with a thread safe strtok. Standard C does not do threads, (up until C11 that is).

A more convenient alternative to strtok is strsep imo, it's also thread safe.

If your program is single threaded then strtok is fine, the potential problem is if several threads use strtok at the same time since strtok use a static variable to keep track of it's progress.

chown33
Mar 6, 2013, 10:07 AM
My C books don't mention strtok_r. Where can I find out about it and what header has to be #included?

Read the man page:
man strtok_r
Or look it up in Xcode's builtin docs.

Or google it.


In a real world application if you read a file line by line from a disk it will go slowly particularly if it's a big file because each command to read or write is a transaction and will take 10 or 15 milliseconds. Your drive can send you a big burst of continuous data rapidly if it's only one transaction.

There's no speed difference for small vs. large files. If you think otherwise, please provide some evidence or explanation.

You should also look at how buffering works in stdio (Standard I/O), because it has a significant affect on how fast data is read or written. You can run tests with buffered vs. unbuffered to see and measure the difference. Hint: It may not be as slow as you think.

Reading a file entirely into memory isn't always a viable approach. Even when it's viable, it may be unnecessary. The OP wasn't having a problem with speed, but with correctness. There's no point in trying to solve speed problems until all the correctness problems have been solved first.