Shocking study reveals brain scan results look different if they are viewed on a Mac"

Discussion in 'Current Events' started by atlanticza, Jun 19, 2012.

  1. atlanticza macrumors 6502a

    atlanticza

    Joined:
    Jul 18, 2008
    Location:
    Cape Town
    #1
    From the UK's Daily Mail:

    Software used to analyse medical brain images gives significantly different results if it’s run on a Apple Mac or PC, according to new research.

    The findings by a team of German researchers has raised concerns
    about potential problems with software analysis.

    Data from 30 brain scans was fed into the computer using a package called FreeSurfer.


    Read more here
     
  2. iJohnHenry macrumors P6

    iJohnHenry

    Joined:
    Mar 22, 2008
    Location:
    On tenterhooks
    #2
    I will hold my shock until it is determined if this is better, or worse, in the treatment of patients. ;)
     
  3. Rodimus Prime macrumors G4

    Rodimus Prime

    Joined:
    Oct 9, 2006
    #3
    The fact that a update in OSX caused such a huge change is an issue. The other one can be correct for but an update to an OS should never cause such a massive change. The question comes is what the hell is Apple screwing with during the updates.
     
  4. chown33 macrumors 604

    Joined:
    Aug 9, 2009
    #4
    Following links through to the original data, we read:
    Abstract
    FreeSurfer is a popular software package to measure cortical thickness and volume of neuroanatomical structures. However, little if any is known about measurement reliability across various data processing conditions. Using a set of 30 anatomical T1-weighted 3T MRI scans, we investigated the effects of data processing variables such as FreeSurfer version (v4.3.1, v4.5.0, and v5.0.0), workstation (Macintosh and Hewlett-Packard), and Macintosh operating system version (OSX 10.5 and OSX 10.6). Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier versions. These differences were on average 8.8±6.6% (range 1.3-64.0%) (volume) and 2.8±1.3% (1.1-7.7%) (cortical thickness). About a factor two smaller differences were detected between Macintosh and Hewlett-Packard workstations and between OSX 10.5 and OSX 10.6. The observed differences are similar in magnitude as effect sizes reported in accuracy evaluations and neurodegenerative studies.The main conclusion is that in the context of an ongoing study, users are discouraged to update to a new major release of either FreeSurfer or operating system or to switch to a different type of workstation without repeating the analysis; results thus give a quantitative support to successive recommendations stated by FreeSurfer developers over the years. Moreover, in view of the large and significant cross-version differences, it is concluded that formal assessment of the accuracy of FreeSurfer is desirable.
    Underlines added for emphasis.

    Summary: the largest differences were seen between FreeSurfer version v5.0.0 and its two earlier versions. Differences about half that size (factor of two smaller) were seen between Mac OS X 10.5 and 10.6 and H-P workstations (the H-P OS version or versions is unspecified in the article abstract, turns out to be CentOS 5.3 in complete article).

    So I agree with the conclusion of the abstract: a formal assessment of FreeSurfer's accuracy is called for. Speaking as a developer, I do wonder what the FreeSurfer developers might have done that lead to this degree of difference.

    And frankly, my shock is reserved for the apparent complete lack of formal accuracy assessment in medical software, free or not.

    Complete free pubmed article link:
    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3365894/?tool=pubmed
     
  5. Mousse macrumors 68000

    Mousse

    Joined:
    Apr 7, 2008
    Location:
    Flea Bottom, King's Landing
    #5
    Maybe it's just me, but... version 4.2.1 on the PC and version 5.0 on the Mac? Different results. And this was unexpected?:confused: Maybe if the ran the same version #'s the difference would be less newsworthy.:rolleyes:
     
  6. Dave00 macrumors 6502a

    Joined:
    Dec 2, 2003
    Location:
    Pittsburgh
    #6
    Nothing to see here. You can't compare results between different versions of the software - says so right on the software web site. There are many variables you can adjust on an MRI, CT, or PET scan that make them look very different based on the same data set - different thresholds, etc. That is probably all you're seeing.

    Move along. :)
     
  7. AnimaLeo macrumors 6502

    Joined:
    Sep 2, 2009
    #7
    For the benefit of you American's: The Daily Mail is not considered a 'serious' newspaper and is often filled with 'shocking' headlines just designed to make you buy the paper.
     
  8. niuniu macrumors 68020

    niuniu

    Joined:
    Mar 29, 2009
    Location:
    A man of the people. The right sort of people.
    #8
    Too many variables.. not ready to be news yet..
     
  9. chrono1081 macrumors 604

    chrono1081

    Joined:
    Jan 26, 2008
    Location:
    Isla Nublar
    #9
    Thank you for that. I was not aware. So its kind of like our Weekly World News? (Although I'd assume not quite as extreme).
     
  10. AnimaLeo macrumors 6502

    Joined:
    Sep 2, 2009
    #10
    I'm not too sure I'm afraid. I think it might be a bit like FOX news. So the news is kinda skewed from a right-wing perspective, very anti-immergration etc etc.
     
  11. andiwm2003 macrumors 601

    andiwm2003

    Joined:
    Mar 29, 2004
    Location:
    Boston, MA
    #11
    That article is lousy and written by a moron who doesn't know what he is talking about. It's a common problem that scientific analysis software can give different results between versions unless it is validated properly. So nothing new here and nothing that has anything to do with the OS per se.

    I'm just surprised that analysis software for brain scan doesn't seem to be certified by the FDA or similar authorities. That would ensure consitency.
     
  12. rhett7660, Jun 21, 2012
    Last edited: Jun 21, 2012

    rhett7660 macrumors G4

    rhett7660

    Joined:
    Jan 9, 2008
    Location:
    Sunny, Southern California
    #12
    Maybe it is because they are running two different versions of the software? That could never ever have been the issue? :rolleyes: It could never be the developer of the software screwing things up could it?
     
  13. Huntn macrumors G5

    Huntn

    Joined:
    May 5, 2008
    Location:
    The Misty Mountains
    #13
    Ah, like the the Enquirer in the States?... :)
     
  14. sammich macrumors 601

    sammich

    Joined:
    Sep 26, 2006
    Location:
    Sarcasmville.
    #14
    Hmm, maybe that's why House and his team diagnosed people so well: because they were using Mac's.
     
  15. iJohnHenry macrumors P6

    iJohnHenry

    Joined:
    Mar 22, 2008
    Location:
    On tenterhooks
    #15
    Well, if they are wrong in their diagnosis, they can always cut open the resulting corpse, and learn from that. :p
     
  16. Dr McKay macrumors 68040

    Dr McKay

    Joined:
    Aug 11, 2010
    Location:
    Kirkland
    #16
    You lie, I'm staying in my Bunker before the immigrants flood in and destroy the country, shortly before the UK becomes completely submerged in water after Global Warming.
     
  17. eRondeau macrumors 6502a

    eRondeau

    Joined:
    Mar 3, 2004
    Location:
    Canada's South Coast
    #17
    People BUY newspapers???
     
  18. AnimaLeo macrumors 6502

    Joined:
    Sep 2, 2009
    #18
    I've never bought one in my life. But I'm only 20.
     
  19. whooleytoo macrumors 603

    whooleytoo

    Joined:
    Aug 2, 2002
    Location:
    Cork, Ireland.
    #19
    "Extra Extra! Software has bugs. Even critical medical software."

    The title is just a bit sensationalist; though the underlying problem is quite worrying. Medical software - and the underlying frameworks and libraries it relies upon - have bugs.

    ----------

    Ooooh..... look at you with your "bunker"! I bet you can afford that because of ripping off the social welfare system, or being a nasty tax dodger; laughing all the way to the bank! ;)
     
  20. Carlanga macrumors 604

    Carlanga

    Joined:
    Nov 5, 2009
    #20
    Here is the full official paper on it:
    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3365894/?tool=pubmed

    and it's conclusion:

    The general conclusion from the present study and the practical advice it occasions is that users of FreeSurfer should exercise caution and restraint before applying a major upgrade in either the FreeSurfer (in particular) or OS version or to switch to a different type of workstation in the context of an ongoing study. This may be a truism and consistent with sound methodology for scientific experimentation and therefore a matter of common sense. However, the numerous questions about this issue posted by the user community seem to demonstrate the opposite and the results presented here reliably quantify the possible consequences. The message of caution applies not only to FreeSurfer but likely may be generalised to other intricate processing packages in the field of neuroimaging. The packages become more and more complex and therefore it is difficult to keep a check on propagation effects resulting from (small) modifications regarding one of the underlying algorithms.

    An additional message inferred from the present study is that authors reporting on results obtained with FreeSurfer are highly recommended to provide not only the version of FreeSurfer that was used but also details on the OS version and workstation.

    Finally, given the large and significant differences between the latest version v5.0.0 and earlier versions, it is concluded that an assessment of the accuracy of FreeSurfer is desirable.
     
  21. iJohnHenry macrumors P6

    iJohnHenry

    Joined:
    Mar 22, 2008
    Location:
    On tenterhooks
    #21
    Some QA, or Model Office, would be appropriate, given that people's lives are at risk.
     
  22. Tsuchiya macrumors 68020

    Tsuchiya

    Joined:
    Jun 7, 2008
    #22
    AnimaLeo is just being polite. The Daily Mail is is a pile of **** that should be avoided.
     
  23. bartelby macrumors Core

    Joined:
    Jun 16, 2004
    #23
    Ammended for accuracy.:)
     
  24. rbrian macrumors 6502a

    rbrian

    Joined:
    Jul 24, 2011
    Location:
    Aberdeen, Scotland
    #24
    This website gives you a sample of the kind of headlines you'll find in the Daily (hate)Mail: http://www.qwghlm.co.uk/toys/dailymail/

    I'm pretty sure the writers use this, and then concoct a mostly made-up story around the headline.
     
  25. iJohnHenry macrumors P6

    iJohnHenry

    Joined:
    Mar 22, 2008
    Location:
    On tenterhooks
    #25
    Adverts drive the news, not the other way around.

    Same with TV.
     

Share This Page