Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Shocking study reveals brain scan results look different if they are viewed on a Mac"

atlanticza

atlanticza

macrumors 6502a
Original poster
Jul 18, 2008
563
2
Cape Town
From the UK's Daily Mail:

Software used to analyse medical brain images gives significantly different results if it’s run on a Apple Mac or PC, according to new research.

The findings by a team of German researchers has raised concerns
about potential problems with software analysis.

Data from 30 brain scans was fed into the computer using a package called FreeSurfer.

Read more here
 
The fact that a update in OSX caused such a huge change is an issue. The other one can be correct for but an update to an OS should never cause such a massive change. The question comes is what the hell is Apple screwing with during the updates.
 
atlanticza said:
From the UK's Daily Mail:

Software used to analyse medical brain images gives significantly different results if it’s run on a Apple Mac or PC, according to new research.

The findings by a team of German researchers has raised concerns
about potential problems with software analysis.

Data from 30 brain scans was fed into the computer using a package called FreeSurfer.

Read more here
Click to expand...

Following links through to the original data, we read:
Abstract
FreeSurfer is a popular software package to measure cortical thickness and volume of neuroanatomical structures. However, little if any is known about measurement reliability across various data processing conditions. Using a set of 30 anatomical T1-weighted 3T MRI scans, we investigated the effects of data processing variables such as FreeSurfer version (v4.3.1, v4.5.0, and v5.0.0), workstation (Macintosh and Hewlett-Packard), and Macintosh operating system version (OSX 10.5 and OSX 10.6). Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier versions. These differences were on average 8.8±6.6% (range 1.3-64.0%) (volume) and 2.8±1.3% (1.1-7.7%) (cortical thickness). About a factor two smaller differences were detected between Macintosh and Hewlett-Packard workstations and between OSX 10.5 and OSX 10.6. The observed differences are similar in magnitude as effect sizes reported in accuracy evaluations and neurodegenerative studies.The main conclusion is that in the context of an ongoing study, users are discouraged to update to a new major release of either FreeSurfer or operating system or to switch to a different type of workstation without repeating the analysis; results thus give a quantitative support to successive recommendations stated by FreeSurfer developers over the years. Moreover, in view of the large and significant cross-version differences, it is concluded that formal assessment of the accuracy of FreeSurfer is desirable.
​
Underlines added for emphasis.

Summary: the largest differences were seen between FreeSurfer version v5.0.0 and its two earlier versions. Differences about half that size (factor of two smaller) were seen between Mac OS X 10.5 and 10.6 and H-P workstations (the H-P OS version or versions is unspecified in the article abstract, turns out to be CentOS 5.3 in complete article).

So I agree with the conclusion of the abstract: a formal assessment of FreeSurfer's accuracy is called for. Speaking as a developer, I do wonder what the FreeSurfer developers might have done that lead to this degree of difference.

And frankly, my shock is reserved for the apparent complete lack of formal accuracy assessment in medical software, free or not.

Complete free pubmed article link:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3365894/?tool=pubmed
 
Maybe it's just me, but... version 4.2.1 on the PC and version 5.0 on the Mac? Different results. And this was unexpected?:confused: Maybe if the ran the same version #'s the difference would be less newsworthy.:rolleyes:
 
Nothing to see here. You can't compare results between different versions of the software - says so right on the software web site. There are many variables you can adjust on an MRI, CT, or PET scan that make them look very different based on the same data set - different thresholds, etc. That is probably all you're seeing.

Move along. :)
 
For the benefit of you American's: The Daily Mail is not considered a 'serious' newspaper and is often filled with 'shocking' headlines just designed to make you buy the paper.
 
AnimaLeo said:
For the benefit of you American's: The Daily Mail is not considered a 'serious' newspaper and is often filled with 'shocking' headlines just designed to make you buy the paper.
Click to expand...

Thank you for that. I was not aware. So its kind of like our Weekly World News? (Although I'd assume not quite as extreme).
 
chrono1081 said:
Thank you for that. I was not aware. So its kind of like our Weekly World News? (Although I'd assume not quite as extreme).
Click to expand...

I'm not too sure I'm afraid. I think it might be a bit like FOX news. So the news is kinda skewed from a right-wing perspective, very anti-immergration etc etc.
 
That article is lousy and written by a moron who doesn't know what he is talking about. It's a common problem that scientific analysis software can give different results between versions unless it is validated properly. So nothing new here and nothing that has anything to do with the OS per se.

I'm just surprised that analysis software for brain scan doesn't seem to be certified by the FDA or similar authorities. That would ensure consitency.
 
Rodimus Prime said:
The fact that a update in OSX caused such a huge change is an issue. The other one can be correct for but an update to an OS should never cause such a massive change. The question comes is what the hell is Apple screwing with during the updates.
Click to expand...

Maybe it is because they are running two different versions of the software? That could never ever have been the issue? :rolleyes: It could never be the developer of the software screwing things up could it?
 
Last edited:
AnimaLeo said:
For the benefit of you American's: The Daily Mail is not considered a 'serious' newspaper and is often filled with 'shocking' headlines just designed to make you buy the paper.
Click to expand...

You lie, I'm staying in my Bunker before the immigrants flood in and destroy the country, shortly before the UK becomes completely submerged in water after Global Warming.
 
"Extra Extra! Software has bugs. Even critical medical software."

The title is just a bit sensationalist; though the underlying problem is quite worrying. Medical software - and the underlying frameworks and libraries it relies upon - have bugs.

----------
Benjy91 said:
You lie, I'm staying in my Bunker before the immigrants flood in and destroy the country, shortly before the UK becomes completely submerged in water after Global Warming.
Click to expand...

Ooooh..... look at you with your "bunker"! I bet you can afford that because of ripping off the social welfare system, or being a nasty tax dodger; laughing all the way to the bank! ;)
 
Here is the full official paper on it:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3365894/?tool=pubmed

and it's conclusion:

The general conclusion from the present study and the practical advice it occasions is that users of FreeSurfer should exercise caution and restraint before applying a major upgrade in either the FreeSurfer (in particular) or OS version or to switch to a different type of workstation in the context of an ongoing study. This may be a truism and consistent with sound methodology for scientific experimentation and therefore a matter of common sense. However, the numerous questions about this issue posted by the user community seem to demonstrate the opposite and the results presented here reliably quantify the possible consequences. The message of caution applies not only to FreeSurfer but likely may be generalised to other intricate processing packages in the field of neuroimaging. The packages become more and more complex and therefore it is difficult to keep a check on propagation effects resulting from (small) modifications regarding one of the underlying algorithms.

An additional message inferred from the present study is that authors reporting on results obtained with FreeSurfer are highly recommended to provide not only the version of FreeSurfer that was used but also details on the OS version and workstation.

Finally, given the large and significant differences between the latest version v5.0.0 and earlier versions, it is concluded that an assessment of the accuracy of FreeSurfer is desirable.
 
Carlanga said:
Finally, given the large and significant differences between the latest version v5.0.0 and earlier versions, it is concluded that an assessment of the accuracy of FreeSurfer is desirable.
Click to expand...

Some QA, or Model Office, would be appropriate, given that people's lives are at risk.
 
AnimaLeo said:
For the benefit of you American's: The Daily Mail is not considered a 'serious' newspaper and is often filled with 'shocking' headlines just designed to make you buy the paper.
Click to expand...

AnimaLeo is just being polite. The Daily Mail is is a pile of **** that should be avoided.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back