CPU core issue?

[mrkgoo]

I have a quad core iMac.

I recently did a restore from Time Machine, and afterwards, would leave the machine to rebuild a large thumbnail database.

However, it has been hanging after a while, with the system taking up a constant 25% of CPU usage, when NO processes take up any observable amount in activity monitor.

Even when I'm forced to Force Quit this job, the system remains at 25% usage (by system), with 75% idle, and no process accounting for this usage until I restart.


Can a core go 'bad'?

I've tried reinstalling Mac OS X AND reinstalling my app, but this still happens, where it didn't used to happen.

 

[maflynn]

However, it has been hanging after a while, with the system taking up a constant 25% of CPU usage, when NO processes take up any observable amount in activity monitor.

Even when I'm forced to Force Quit this job, the system remains at 25% usage (by system), with 75% idle, and no process accounting for this usage until I restart.


Can a core go 'bad'?

Yes, in theory a core can go south, but by the same token, an application has to be written to take advantage of multiple cores, i.e., separating work units into threads. Some tasks can easily be split some cannot. Just because you're seeing an application not take an advantage of multiple cores does not mean something is wrong.

I would thing you would be receiving system warning at boot up and/or during operation that a core has gone bad if it was defective.

 

[xraydoc]

Also, with Activity Monitor, make sure you're looking at ALL the system processes - not just the ones running as you as the user. If you can't see what's using the CPU, you may be looking only at your user account processes and not all the system processes.

 

[mrkgoo]

Yes I'm most definitely looking at all processes.

Basically what happens is my computer locks up during a process requiring force quitting. I notice that when it locks up, 25 % of my CPU (or one core equivalent) is used by 'system' (it's red on the monitor and says 25% system use, 75% idle). And I need to force quit the application (aperture).

Even when I force quit, aperture stays on the proces list, but drops to zero (not sure if this is normal or not, could be part of how apps suspending feature worked since
ML).

The processor at 25% remains and I can't open that application until I restart.

Hmmm, I might try a diagnostics on the hardware.

 

[xraydoc]

Yes I'm most definitely looking at all processes.

Basically what happens is my computer locks up during a process...

What process? The same one every time, or random apps are crashing, causing Aperture to seize up? I'm not an Aperture user, so I don't know if it leave something running in the background at all times or not, but if it's a consistently reproducible error, it sounds like a software issue to me. Perhaps an old version of a 3rd party extension or driver causing a problem after you re-installed everything?

Try a clean install of the OS and putting Aperture on a virgin system and seeing if you get the same issue. If not, then add your apps back one at a time, carefully testing after each one, until you find the culprit. If it fails on a fresh, clean install of the OS with nothing else installed, or if it's a completely unpredictable error, then it may be a RAM or logic board problem.

 

[mrkgoo]

That's just it - I can't see the process. Are there such things as invisible or hidden processes? Like would "top" or "ps" in terminal help me here?

Activity monitor simply sits at 25% used by system, 75% idle. I don't know if it's a cause or effect of something that happens while Aperture is rebuilding thumbnails (thumbnails don't get backed up by Time Machine, hence why it's rebuilding them after a restore).


Yeah I get that it might be something third party, but I'm generally very clean. I bought the mac last year and started it clean and just transferred my user data. Sure, my user data has years of fluff from being transferred about, but I had absolutely no issues during the past year like this having even done a couple of restores, just not on Mavericks.

I know it's the systematic approach to go one bye one eliminating apps, but generally those cases are so inconvenient. Like what am I supposed to do? Erase and clean install and put just my aperture library in from time machine and watch it rebuild for hours to see it happen again or not, then delete it, add one component, replace the library and observe again?

It takes hours just to move my library and hours during a rebuild before a fail happens. It could take weeks or months of solid work time to resolve. Nuts!

The error could be anywhere! A line of code in any of the hundreds of preference lists. Could be baked into Mavericks, like a hardware driver. Could be hardware!

It already takes hours to do even a clean install and restore the aperture library. And what gets fine then? It's just testing. I can't start a slow rebuild of system component by component until I find the culprit, because even just restoring a library and then opening it will change it such that it's not the same. That's the problem with such right integration of apple software - it's great for when it just works, but when t doesn't, it can be a true nightmare.

Like piecing back my system bit by bit means it will not actually be the same - I will inadvertently break links to other things and probably damage them beyond my knowledge to repair. For example, restoring my library without metadata flags might irreversibly mess up the 50,000 files that are organised.

Basically it is a huge pain to try and systematically eliminate the problem because of the hundreds if not thousands of components that make up a system and could be the issue combined with each testing step utilising dozens of hours.

I guess I was just hoping someone would recognise the symptoms and help
Me zero in on possible issues. Thank for your help.

 

[xraydoc]

So how do you know a process has crashed then? Because of the 25% CPU usage? Are you certain that it's not a legitimate thing it's doing? Have you let it go for an hour or two to see what happens?

And you don't need to erase Aperture every single time. Install OS, install Aperture and your library. Run Aperture and look for your odd CPU usage. If none, install next app. Look again. Install next app and look again.....

And you think this is a long process? Try doing this on a Windows system!

 

[maflynn]

]
The processor at 25% remains and I can't open that application until I restart.

Hmmm, I might try a diagnostics on the hardware.

That's not right and that doesn't sound like a hardware issue. Back up your data and go with a clean install as xraydoc stated. I use Aperture and my rMBP does not lock up at all when using it.

If I force quit any apps including aperture, I can restarted them fairly quickly. Time to reinstall OSX to see if that corrects the issue.

 

[mrkgoo]

So how do you know a process has crashed then? Because of the 25% CPU usage? Are you certain that it's not a legitimate thing it's doing? Have you let it go for an hour or two to see what happens?

And you don't need to erase Aperture every single time. Install OS, install Aperture and your library. Run Aperture and look for your odd CPU usage. If none, install next app. Look again. Install next app and look again.....

And you think this is a long process? Try doing this on a Windows system!

Well, it may not be apps. It could be a rogue preference file. It could be a corrupt library. The reason it would take a long time is that my library is 600 GB. reinstalling it from a backup to troubleshoot would take forever - and yes, I have to reinstall the Aperture Library every time to be consistent - as it's the thumbnail rebuild that shows the issue after a few hours (again this adds to the trouble shoot time). I guess technically, I COULD just delete thumbnails, but you know, to be properly consistent.

ie - Clean install of OS X - 3-4 hours (internet connection not the best) - reinstall Aperture and Library 3-4 hours. Let thumb nails build until it "hangs" - 2-3 hours.

Then add new test-addition - then either reinstall aperture library and let thumnails build again, etc etc.

It's not like it's a small library, with a few minutes of building to see the hang.

At any rate, I reinstalled OS X and Aperture, and it was BETTER, but not fixed. It happened less frequently, and allowed me to just brute force the rebuild. the "hang" ONLY happened during thumbnail rebuilds.

No, I didn't let it go for several hours, but probably for an hour at least. The weird thing was that it was a SYSTEM thing, but no associated process. I posted this on an Aperture-related board, and one person chimed in and said they've seen the same thing, so that would suggest a software bug (again I haven't done a large thumbnail rebuild on Mavericks an the latest Aperture).

That's not right and that doesn't sound like a hardware issue. Back up your data and go with a clean install as xraydoc stated. I use Aperture and my rMBP does not lock up at all when using it.

If I force quit any apps including aperture, I can restarted them fairly quickly. Time to reinstall OSX to see if that corrects the issue.

Aperture doesn't lock up when I'm using it. It locks up during thumbnail rebuilding. 50,000 thumbnails, it gets through a few thousand with a user process (green) then just stops, and a red process (belongs to system) uses exactly 25% and just sits there, even if I force quit Aperture.

I know it's not normal, I was mostly just curious if anyone else has seen this to get an idea if it's unique to my setup or just a rare instance. It helps to troubleshoot before going on an epic journey to bit-by-bit test.

I simply don't have the time to spend on piecing together my system from scratch, unless it gives me a really good lead to where the issue is. I know the very basic step to take would be clean install and add just the library - that would tell me if it's a library and/or software issue. But yeah, it's already several days for my system to be rebuilt if I'm working as best I can, let alone having to piece it together to test.

But like I said, I have brute forced the the thumbnail rebuild. No doubt there is SOME issue SOMEWHERE. at a guess I would say it's either the interaction with Mavericks and Aperture (a bug that only comes up in my particular instance i.e with my system configuration), or it's my Aperture Library itself (it has quirks since it has been built from early versions and simply ported) - I've been meaning to export it into a new library to start anew, but frankly, it's a case of "if it's not severely broken - don't fixt it", because it's such a huge library, any rebuild is not likely to go smoothly, and I'd need to check it properly to see if things move smoothly.

One day, when I have the time and inclination.

Thanks for the input, though, greatly appreciated.

 

[dollystereo]

I would recommend spliting your library in smaller libraries, having a 600gb library is a nonsense.
Aperture keeps everything inside it's library folder, so spliting it, will help you to have speed-up and find you desired problem. Maybe you have some corrupt pictures inside.
I would recommend library per event basis (maybe year, project, or type). So you will have many libraries, but smaller ones.

 

[mrkgoo]

I would recommend spliting your library in smaller libraries, having a 600gb library is a nonsense.
Aperture keeps everything inside it's library folder, so spliting it, will help you to have speed-up and find you desired problem. Maybe you have some corrupt pictures inside.
I would recommend library per event basis (maybe year, project, or type). So you will have many libraries, but smaller ones.

Yeah, I've heard some people complain about Aperture and large libraries.

I wouldn't say it's nonsense though. It's a pro program, right? Hell, some pros take more photos than in my entire library in a year!

I think splitting it up would likely fix the issue, if it's library-related, if ONLY because I'm creating new libraries. I'm willing to bet if I import the entire library into a single new one, that would resolve it too *IF* it's a library-related issue.

I've been in contact with Aperture engineers in the past, and this kind of tactic pretty much ensures library-issues are fixed, because you're essentially starting from zero with a library layout designed for the current version of the software. I've probably got too many hanger-ons from past versions.

But at the moment, it's a non-issue now that my thumbnails are built. I'll keep it in mind when I finally find the time to sit down and do something about it.

Cheers, thanks.

 

[dollystereo]

Yeah, I've heard some people complain about Aperture and large libraries.

I wouldn't say it's nonsense though. It's a pro program, right? Hell, some pros take more photos than in my entire library in a year!

I think splitting it up would likely fix the issue, if it's library-related, if ONLY because I'm creating new libraries. I'm willing to bet if I import the entire library into a single new one, that would resolve it too *IF* it's a library-related issue.

I've been in contact with Aperture engineers in the past, and this kind of tactic pretty much ensures library-issues are fixed, because you're essentially starting from zero with a library layout designed for the current version of the software. I've probably got too many hanger-ons from past versions.

But at the moment, it's a non-issue now that my thumbnails are built. I'll keep it in mind when I finally find the time to sit down and do something about it.

Cheers, thanks.

I dont remember where I found a very good Aperture workflow, and It said specifically to split libraries. I have been doing so for year now, with libraries between 1Gb and 10GB, and it flyes!
Give it a try, 600GB is a huge amount of data, maybe a small split of 6x100GB?

 

[joema2]

...It could be a rogue preference file. It could be a corrupt library....my library is 600 GB....it was a SYSTEM thing, but no associated process...that would suggest a software bug...It locks up during thumbnail rebuilding. 50,000 thumbnails...then just stops, and a red process (belongs to system) uses exactly 25% and just sits there, even if I force quit Aperture....

I don't use Aperture, but I see other photographers with Aperture libraries larger than that having no problems.

If the Aperture process hangs it should be visible in Activity Monitor. If you "force quit" the process and global CPU remains at 25%, the individual process responsible should be visible in AM. Click on the CPU button, click the %CPU column heading (sort list of processes in descending order by by CPU usage).

The Aperture library is a database. Sometimes database access code makes assumptions about data integrity without doing validation. E.g, assume a pointer is good, dereference it and run a procedure, which then gets stuck in a CPU loop because of bad data.

The Aperture developers are responsible for the integrity of their own database -- nobody is accessing it except their software. Due to non-ideal real world effects (file system problem, etc) it's possible a lower layer problem could induce an error in a higher-level database structure. This is why serious databases have internal checking utilities -- much like chkdsk or fsck except at a higher conceptual layer which understands the database-specific internal structures.

There may be an Aperture utility to check or rebuild the library database.

 

[mrkgoo]

I dont remember where I found a very good Aperture workflow, and It said specifically to split libraries. I have been doing so for year now, with libraries between 1Gb and 10GB, and it flyes!
Give it a try, 600GB is a huge amount of data, maybe a small split of 6x100GB?

Yeah I'll consider it for when I do do a library overhaul. Thanks!

I don't use Aperture, but I see other photographers with Aperture libraries larger than that having no problems.

If the Aperture process hangs it should be visible in Activity Monitor. If you "force quit" the process and global CPU remains at 25%, the individual process responsible should be visible in AM. Click on the CPU button, click the %CPU column heading (sort list of processes in descending order by by CPU usage).

The Aperture library is a database. Sometimes database access code makes assumptions about data integrity without doing validation. E.g, assume a pointer is good, dereference it and run a procedure, which then gets stuck in a CPU loop because of bad data.

The Aperture developers are responsible for the integrity of their own database -- nobody is accessing it except their software. Due to non-ideal real world effects (file system problem, etc) it's possible a lower layer problem could induce an error in a higher-level database structure. This is why serious databases have internal checking utilities -- much like chkdsk or fsck except at a higher conceptual layer which understands the database-specific internal structures.

There may be an Aperture utility to check or rebuild the library database.

I see, thanks for your input.

I do not see the process in Activity Monitor at all even after Force Quit. That's why it's so odd. It's red in the activity monitor chart, so it's supposedly used by the System.

Anyway, there are internal rebuilds and check functions in Aperture - holding opt-cmd while opening a library gives you 3 options with escalating troubleshooting options:

repair permissions
repair library database
rebuild library database

I used to every now and again do a rebuild of my library. But since upgrading to 3.4-.5, the rebuild actually hangs during faces rebuild and ends up erasing it instead. At some point an Aperture update seems not to be able to handle the faces database. I had to do a time machine restore during my last rebuild, and it was a lot of work to get back to where I am.

IMO, it could be related to my library, in that it has been carried over from several .x releases of aperture, where they've done a lot of transferring old libraries. This is why I think the issue is probably likely to lie there.

I mean, even in the past, there were weird quirks. For example, originally, repairing a library database could result in Time Machine thinking the entire library had changed and attempting to back it up again (you can imagine what this is like with several 100GBs of library). After a couple versions, that was fixed, and I can repair again.

One day I will again try the rebuild option, but only If I have a hunch I can trust the rebuild option, but also time to do a Time Machine restore afterwards in event of failure.

But since I never had thumbnail rebuilding issues before, it could just as easily be a weird quirk with OS or app.

Another example is importing my Aperture albums to my iPhone - used to work really well, but after Mavericks was released, it started taking a VERY long time (like 5 minutes to check my library and decide what to sync). It may have just been fixed in the latest iTunes release - an X.X.1 point release.

Anyway, point is, my library is working for now, so I'm going to stick with this for now until I see anything change. Troubleshooting large libraries like this just takes up too much time that I don't have.

 

[joema2]

...since upgrading to 3.4-.5, the rebuild actually hangs during faces rebuild and ends up erasing it instead....After a couple versions, that was fixed, and I can repair again....One day I will again try the rebuild option, but only If I have a hunch I can trust the rebuild option...

If Rebuild won't work (or deletes data), that implies an internal data integrity problem in the Aperture library database. This could explain the Aperture hangs and 25% CPU state.

Aperture makes SQLite database calls which in turn manages the database. The database code should ideally always validate data. Failure to do so can cause a program hang or crash if the data is less than perfect.

However in reality compromises must be made -- there's a performance cost to validating data at run time and the vast majority of time the data is OK. For this reason extensive database integrity checks are often deferred to an off-line utility.

A database inconsistency is a disagreement between two internal data structures. Unfortunately it's sometimes impossible to know which one is right - one or the other must be picked, and the database structure brought into agreement with the 'winner'. This can result in data loss.

Sometimes the inconsistent structures can be 'sacrificial', just discarded and rebuilt. E.g, a corrupt index. Normally you'd expect the repair/rebuild utility to handle that but they vary in the degree of automation and required user steps. I see a few threads where Aperture users have file-level repairs:

https://discussions.apple.com/message/11683186#11683186
http://archive.bagelturf.com/aparticles/library/libinside/index.php

Despite working most OK, if your Aperture database has internal problems, that potentially jeopardizes further work. Over time, unrepaired database damage tends to propagate or degrade.

Will the "Repair Database" (not Rebuild) option run to completion on your current library with no errors?

 

[mrkgoo]

If Rebuild won't work (or deletes data), that implies an internal data integrity problem in the Aperture library database. This could explain the Aperture hangs and 25% CPU state.

Aperture makes SQLite database calls which in turn manages the database. The database code should ideally always validate data. Failure to do so can cause a program hang or crash if the data is less than perfect.

However in reality compromises must be made -- there's a performance cost to validating data at run time and the vast majority of time the data is OK. For this reason extensive database integrity checks are often deferred to an off-line utility.

A database inconsistency is a disagreement between two internal data structures. Unfortunately it's sometimes impossible to know which one is right - one or the other must be picked, and the database structure brought into agreement with the 'winner'. This can result in data loss.

Sometimes the inconsistent structures can be 'sacrificial', just discarded and rebuilt. E.g, a corrupt index. Normally you'd expect the repair/rebuild utility to handle that but they vary in the degree of automation and required user steps. I see a few threads where Aperture users have file-level repairs:

https://discussions.apple.com/message/11683186#11683186
http://archive.bagelturf.com/aparticles/library/libinside/index.php

Despite working most OK, if your Aperture database has internal problems, that potentially jeopardizes further work. Over time, unrepaired database damage tends to propagate or degrade.

Will the "Repair Database" (not Rebuild) option run to completion on your current library with no errors?

Yes repair appears to work fine.

Anecdotally speaking, it appears that the issue in terms of rebuilding ability (not my issue reported initially for this thread, although the two could very well be related), would be related to faces.

I recall much before Aperture 3.5, perhaps 3.4 or 3.3, rebuilds went from being fast to very slow - the dialogue box would report it was working on faces, and it would take an EXTREMELY long time, like 10 hours. This ONLY happened on a major point release of Aperture, so I assumed it was the software that was at fault, not my library, at least at that time.

I did several rebuilds during that time, and it was only during a relatively recent release that the rebuild stopped working entirely (and only once), and when that rebuild failed, it crashed out and my faces database was either deleted or corrupted. At that point, I simply restored my library from Time Machine.

There have been updates since that point, but I have not built the courage to try a rebuild since then (perhaps it was a MAvericks thing if I recall - at least in timing), so it may very well be fixed by now. OR it may be, as you say, an integrity issue with my library.

I'm very aware, the issue will need to be faced at some point, but as mentioned, this time in my life is NOT that point. When that time arrives, I would try a rebuild, and if that doesn't work, I may simply export into a new library entirely, or some other solution.

Rebuilds, even by Apple's suggestion, are supposed to be done as a last resort (even though I did do them fairly frequently), when running into some issue. The rebuilds, I noticed were actually increasing the library size by several GBs each time, which I assume meant it was moving old database files into an archive area, but without any really knowledge of what it was doing, it meant I had no control over it, and I eventually lost favour of doing rebuilds.

repair permissions and repair database seem to run fine (repair didn't use to work 100% either, as it would conflict with Time Machine , which would try entire library backups and not incremental ones after a repair previously - this is now not the case).

But like I said, while my library is currently working to MOSTLY my satisfaction, I'm want to sort of hold of on any 'major surgery'. Time will come when I will need to, but I will wait for some Aperture updates before that time just to see if any such issues are fixed, just to give myself a better chance of not having to do too much of an overhaul.