Bit rot

1. 2018-03-17 at 8:09 PM UTC

   #1
   The Self Taught Man Black Hole
   Has anyone actually experienced this? By bit rot I mean a bit getting flipped by cosmic radiation or magnetic field degradation on static media, causing data corruption that is undetectable by the filesystem and only noticeable once you try to open the file and everything is garbled.
   
   Some people consider it a serious issue and use checksumming filesystems like ZFS with RAID to detect and repair silent corruption. I've never encountered it on my personal boxes or any servers I've worked on, so I think these concerns may be overblown and the Hamming codes used for error correction on modern drives do a pretty good job of dealing with this at a hardware level. That said, I do store important files in archive formats that can recover from some level of corruption, and I'm considering a switch to btrfs or ZFS on my NAS and running a monthly scrub just in case.
2. 2018-03-17 at 8:17 PM UTC

   #2
   The Self Taught Man Black Hole
   How would you know that was the cause of the problem though?
3. 2018-03-17 at 8:19 PM UTC

   #3
   

   A College Professor victim of incest [your moreover breastless limestone]
   im not saying it was aliens but...
4. 2018-03-17 at 9 PM UTC

   #4
   The Self Taught Man Black Hole

   Originally posted by greenplastic How would you know that was the cause of the problem though?

   
   If you had a known good copy of a file on a drive, then tried to access it later and found it had been corrupted/the checksum had changed, you could be reasonably sure it was bit rot. But you raise a good point. Since most people don't verify successful copying of data using checksums, I think in a lot of cases the corruption they attribute to degradation of data on disk is probably a copying error caused by bad RAM, a buggy SATA controller, etc.
5. 2018-03-17 at 9:15 PM UTC

   #5
   

   infinityshock Black Hole
   you do realize the actual physical residence of the data is on a cheaply made piece of thin metal, embedded within a 'matrix' of magnetic 'goo' with the resiliency of a snow man in death-valley...rite?
6. 2018-03-17 at 9:16 PM UTC

   #6
   

   benny vader YELLOW GHOST
   i use teracopy with copy verification.
   
   is that good enough ???
7. 2018-03-17 at 9:20 PM UTC

   #7
   The Self Taught Man Black Hole

   Originally posted by benny vader i use teracopy with copy verification.
   
   is that good enough ???

   
   For maintaining integrity during copying, sure. Degradation on disk is a different issue.
8. 2018-03-17 at 10:36 PM UTC

   #8
   

   benny vader YELLOW GHOST

   Originally posted by inb4l0pht For maintaining integrity during copying, sure. Degradation on disk is a different issue.

   
   i dont think theres a way to prevent data degradation of disks.
   
   thats the difference between an expensive disk and a cheap one.
   
   i used to archive all my datas into DVDs but eversince my HDD size went over 100Gb thats no longer possible.
9. 2018-03-17 at 11:25 PM UTC

   #9
   

   -SpectraL coward [the spuriously bluish-lilac bushman]
   This used to happen all the time with compressed Microsoft .cab files. One minute the file would be fine, next minute it would be corrupt. No rhyme or reason.
10. 2018-03-19 at 8:36 AM UTC

    #10
    

    Lanny Bird of Courage
    Surprisingly the rate of cosmic radiation bit flipping (one of my favorite metrics) is actually high enough that most people with a rotational harddrive have probably experienced it a few times. Every filesystem you're likely to use implements CRC or some other kind of error correction logic though, so the chances of you experiencing data loss as a result is so vanishingly low that it's extremely unlikely anyone here has ever seen it. This dude has some math on it.
    
    Spectral is just fucking retarded because there is zero reason for one filetype to be more affected than another by the phenomenon.
    The following users say it would be alright if the author of this post didn't die in a fire!

    • The Self Taught Man
11. 2018-03-19 at 10:49 AM UTC

    #11
    

    -SpectraL coward [the spuriously bluish-lilac bushman]
    .Cab files absorb more cosmic radiation than other filetypes.
12. 2018-03-19 at 10:57 AM UTC

    #12
    

    Lanny Bird of Courage

    Originally posted by -SpectraL .Cab files absorb more cosmic radiation than other filetypes.

    
    I mean this seems like an irony post but it's spectral after all, he could actually be this dumb.
    The following users say it would be alright if the author of this post didn't die in a fire!

    • aldra
13. 2018-03-19 at 11:14 AM UTC

    #13
    

    -SpectraL coward [the spuriously bluish-lilac bushman]
    I've also found .rar files to be quite suspect.
14. 2018-03-19 at 5:55 PM UTC

    #14
    

    benny vader YELLOW GHOST

    Originally posted by -SpectraL .Cab files absorb more cosmic radiation than other filetypes.

    
    true but not becos of the file type but due to how windows OS places files of this type on the HDD.
    
    .cab files are fragmented and spreaded all over the HDD unlike other files that were written in clusters .... and becos of their spread, their chances of getting hit by cosmic radiation were much greater.
    
    the solution to this problem is incredibly simple ... simply mount your HDD vertically, this decreases the chances of your HDD platter getting hit by cosmic rays exponentially.
    The following users say it would be alright if the author of this post didn't die in a fire!

    • -SpectraL
15. 2018-03-19 at 6:01 PM UTC

    #15
    The Self Taught Man Black Hole

    Originally posted by -SpectraL I've also found .rar files to be quite suspect.

    
    That's why I only use .rawr files
    
    
16. 2018-03-19 at 6:51 PM UTC

    #16
    

    -SpectraL coward [the spuriously bluish-lilac bushman]
    If you want to avoid data rot 100%, take the data you want to preserve and compress it into segmented .rar files, then use a program called PAR to make data recovery block files. No matter how corrupted the archived set gets, PAR can reconstruct the data.
    
    http://www.quickpar.org.uk/AboutPAR2.htm
17. 2018-03-19 at 6:56 PM UTC

    #17
    

    benny vader YELLOW GHOST
    sounds like money is needed.
18. 2018-03-19 at 8:29 PM UTC

    #18
    The Self Taught Man Black Hole

    Originally posted by benny vader true but not becos of the file type but due to how windows OS places files of this type on the HDD.
    
    .cab files are fragmented and spreaded all over the HDD unlike other files that were written in clusters …. and becos of their spread, their chances of getting hit by cosmic radiation were much greater.
    
    the solution to this problem is incredibly simple … simply mount your HDD vertically, this decreases the chances of your HDD platter getting hit by cosmic rays exponentially.

    
    I’ve actually found that if you mount your HDD at a 45 degree angle, data corruption due to bit rot is essentially zero. This is because cosmic rays will bounce right off, rather than if they hit straight on
19. 2018-03-19 at 8:31 PM UTC

    #19
    

    Zigzagoon Yung Blood
    This is crazy, didn't know this was even a thing.
    
    So how would one store data for a long period of time like hundreds of years?
20. 2018-03-19 at 8:37 PM UTC

    #20
    The Self Taught Man Black Hole

    Originally posted by Zigzagoon This is crazy, didn't know this was even a thing.
    
    So how would one store data for a long period of time like hundreds of years?

    
    I think I read the most stable form of long term data storage is on CD. You couldn’t just use any optical disc though, you need to use the ones made of polycarbonate or some shit.