Stuck Processes.

[spooner1887]

So I'm been running a c-program that takes rougly 25 to 60 hours to complete. It's purpose is to analyze data sets, and in this case rather large data sets (~1Gb). The Mac I'm running them on has OS X 10.6.6 and has 2 6-Core Intel Xeon Processors.

So I've written short shell scripts to run my program, and they seemed to run fine when I submitted my jobs last night. I get into work this morning and check on how they are doing.

I run "top" and I notice that all three programs are in the STATE "stuck". Every few screen updates they will momentarily go into the STATE "running". But most of the time they are "stuck". I'm concurrently running the same programs (different data), on a Red Hat machine, and they seem to be doing just fine. I've also successfully run my program on much smaller data sets many times in the past. I've also ran it successfully on similarly sized datasets.

I thought that perhaps that they got stuck do to memory issues. However I have 32Gb, with 2.5Gb active, 23Gb inactive, 27Gb used, and 5Gb free.

So my questions:
What are the causes to programs getting "stuck"?
How do I prevent programs from getting "stuck"?
Is there a solution to get programs 'unstuck'?

Killing the processes does not seem like a good option. If my program has run 20 hours and will take 50+hours to complete, it would be an enormous waste of time to kill them. What would guarantee them from not getting stuck again? Thanks everyone for your time.

~spooner

 

[larkost]

You have not posted either what the programs are, or anything about the shell script you created. So any answer by us is going to have to be a completely wild guess.

So on a completely wild guess: you used Python, specifically the "subprocess" module with "stdout=subproces.PIPE", and your programs have created more data than the buffer can hold, and so they are unable to "write" anymore to the buffer and are now stuck waiting for IO to complete.

If my wild guess is right, then you need to change your stdout option to point at real files (unfortunately StringIO and cStringIO objects don't work).

But far more likely my wild guess is wildly off the mark, and you need to learn to ask better questions.

 

[spooner1887]

I appreciate the "help" that you gave, but the program I am using is not something that is available to anyone outside of about a dozen people, so telling you what it is is not going to be of any use now is it? As far as asking better questions, I could ask it better if I knew what the ******* is going on, now couldn't I? I don't particularly think it would be useful to post 10000 lines of code for you to generously investigate.

As far as my shell script is concerned, it is a short (only 4 lines) bash script that is SOLELY passing the command line options that I would have to normally pass when running my program. Its sole purpose is to run my program when I'm not there. I execute my program 3 times with different options, and this is all my bash script does.

I've searched for about an hour, trying to see what "stuck" processes are and what causes them, but I haven't found any useful information on them. That is why I'm asking about it here.
Thanks again.

 

[maflynn]

My only suggestion is place debugging code in your program and/or checkpoints.

As the other poster mentioned, you're requesting help on a program that none of us knows any details. We cannot tell you why your program is hanging, so you need to set some traces and/or displays inside the code to see what's going on as it executes.

 

[larkost]

I appreciate the "help" that you gave

The sarcasm implied by the quotes is really dumb on your part.

but the program I am using is not something that is available to anyone outside of about a dozen people, so telling you what it is is not going to be of any use now is it?

If you want people troubleshooting it, then yes, knowing what sort of thing it is would give us a better chance of guessing what is going on. Remember, you are coming to us, thus expecting us to be smarter on this subject than you. Keeping us in the dark about what you need help with seem particularly bone-headed on your part.

As far as asking better questions, I could ask it better if I knew what the ******* is going on, now couldn't I?

That does seem to be my point, now doesn't it?

I don't particularly think it would be useful to post 10000 lines of code for you to generously investigate.

Nowhere did I ask for the program code. I asked for the general class of program.

As far as my shell script is concerned, it is a short (only 4 lines) bash script that is SOLELY passing the command line options that I would have to normally pass when running my program. Its sole purpose is to run my program when I'm not there. I execute my program 3 times with different options, and this is all my bash script does.

a) If it is only 4 lines, why have you not posted it?
b) Why is this the way that you are telling us that it is in bash? This is exactly what I was talking about when I said you need to learn to ask better questions. I believe you need to take a solid read (not skim) over this:

http://www.catb.org/~esr/faqs/smart-questions.html

I've searched for about an hour, trying to see what "stuck" processes are and what causes them, but I haven't found any useful information on them. That is why I'm asking about it here.

I have already indicated the #1 reason for "stuck" processes: they have filled up (or emptied out) a buffer. Since when you run without the script you say you don't encounter the problem, that tends to indicate that the script is holding onto said buffer... but since you can't let anyone see your script, or provide any sample code... no one is ever going to be able to help you.