Making Fat Sd Cards More Robust

[daniel3000]

Hi guys,

I have observed, that when you remove a FAT-formatted SD card without unmounting it cleanly, the entire FAT contents may be erased.
Sometimes only the files which have been added to the FS are gone, but sometimtes the card will appear EMPTY after such faulty handling.

Two questions:

Is there a way to make a FAT-formatted SD card more robust, so something like this won't happen? (e.g. switching off some FAT write cache, adding more FAT copies etc. etc.)?

Is there a way to rebuild the FAT (automatically, not manually) after such an incident?

Windows' scandisk won't recover the FAT, and fsck.vfat won't, too, even when using the first OR the second FAT copy.

Thank you for your ideas!

Daniel

[InSearchOf]

well you could change the mount option to async... that may help... instead of sync which is the default

Late

[desertrat]

well you could change the mount option to async... that may help... instead of sync which is the default

But isn't sync supposed to keep the filesystem (hence FAT) updated immediately (at the expense of using extra flash write cycles) whereas async will cache changes to the filesystem until the kernel(?) decides it's time to write the changes or you do sync or umount, whichever is sooner.

[Meanie]

well you could change the mount option to async... that may help... instead of sync which is the default

But isn't sync supposed to keep the filesystem (hence FAT) updated immediately (at the expense of using extra flash write cycles) whereas async will cache changes to the filesystem until the kernel(?) decides it's time to write the changes or you do sync or umount, whichever is sooner.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157263\"][{POST_SNAPBACK}][/a][/div]

yeah, desertrat got it right.

the only way is to properly unmount your partition, and adding an extra sleep in the eject script so it actually waits an additional second or two for the filesystem to sync up completely if it is a larger SD card.

note that the two FAT tables are only applicable if you are using FAT12 or FAT16. FAT32 uses both tables since it needs the extra space to store long file names and support larger filesystems...

[daniel3000]

yeah, desertrat got it right.

the only way is to properly unmount your partition, and adding an extra sleep in the eject script so it actually waits an additional second or two for the filesystem to sync up completely if it is a larger SD card.

note that the two FAT tables are only applicable if you are using FAT12 or FAT16. FAT32 uses both tables since it needs the extra space to store long file names and support larger filesystems...
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157273\"][{POST_SNAPBACK}][/a][/div]

aha, so for a card up to 2GB a FAT16 fs would be better (i.e. safer) than a FAT32, right?

Is the only disadvantage of FAT16 the large cluster size of 32kB and the fixed root directory size?

daniel

[daniel3000]

note that the two FAT tables are only applicable if you are using FAT12 or FAT16. FAT32 uses both tables since it needs the extra space to store long file names and support larger filesystems...
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157273\"][{POST_SNAPBACK}][/a][/div]

On a German website (don#t remember where, though) I got other information.
There it was written "FAT32 uses, as opposed to FAT16, the second FAT really for error correction".
which seems to imply that the second FAT in FAT16 was simply redundant and unused...

Which information is correct?
Do you have a good source for correct information?
There is not much information about that on the internet.

thanks

daniel

[InSearchOf]

well you could change the mount option to async... that may help... instead of sync which is the default

But isn't sync supposed to keep the filesystem (hence FAT) updated immediately (at the expense of using extra flash write cycles) whereas async will cache changes to the filesystem until the kernel(?) decides it's time to write the changes or you do sync or umount, whichever is sooner.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157263\"][{POST_SNAPBACK}][/a][/div]

But if Daniel was stating that removing a sd card on the fly instead of umount is where the issue is caused... this is the exact reason why floppy drives are mounted sync because a typical person doesnt umount mount a floppy before removing it.

Sync will allow a user to remove there sd with out the hassle of umount'in it but yes you will suffer when it comes to write counts on a "write limited" device...

Late

[daniel3000]

But if Daniel was stating that removing a sd card on the fly instead of umount is where the issue is caused... this is the exact reason why floppy drives are mounted sync because a typical person doesnt umount mount a floppy before removing it.

Sync will allow a user to remove there sd with out the hassle of umount'in it but yes you will suffer when it comes to write counts on a "write limited" device...

Late
[div align=\"right\"][{POST_SNAPBACK}][/a][/div]

Just found this text on Microsoft TechNet (and they should know well about their FSs):

Code: [Select]

FAT32 is more robust than FAT16. FAT32 has the ability to relocate the root directory and use the backup copy of the FAT instead of the default copy. In addition, the boot record on FAT32 drives has been expanded to include a backup of critical data structures. This means that FAT32 volumes are less susceptible to a single point of failure than FAT16 volumes.
It's found here:
[a href=\"http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/core/fncc_fil_blpd.mspx?mfr=true]http://www.microsoft.com/technet/prodtechn...d.mspx?mfr=true[/url]

Well, this is the theory and probably true in Windows.
but is it also true in Linux with the vfat fs driver?
The driver may be old and thus support FAT16 better than FAT32.

Who has experiences or even knowledge?


Regarding the sync mount:

Is sync the default?
in /etc/sdcontrol the line "FATOPTS=..." is commented out, and it even doesn't contain sync or async, so there is no parameter "sync" for the SD card mount command.
When looking at the mout table, there is no mention of "sync".

Also, as InSearchOf says, the mout man page says:

Code: [Select]

All I/O to the file system should be done synchronously. In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening
so actually from this point of view, sync is not good for flash cards.
However, I would rather be failsafe now than in the future because I expect to exchange my SD card anyway after some time.
Some even have swap on SD and say it works for months without a problem.
So lifeycle is probably not a big issue anymore.

Is there any known value in how mucht "sync" will increase the number of write processes? Probably depends heavily on the kind of usage. But typical usage, i.e. open a file, work on it, save it, will not let "sync" increase write cycles much, would it?

Maybe with a swap file on SD it would. Or with other random access type files.

What's your opinion?



daniel

[daniel3000]

Here is a very interesting article, discussing several aspects about mounting sync:

https://wiki.ubuntu.com/SyncMount

I think I will try it. :-)

daniel

[Meanie]

Code: [Select]

FAT32 is more robust than FAT16. FAT32 has the ability to relocate the root directory and use the backup copy of the FAT instead of the default copy. In addition, the boot record on FAT32 drives has been expanded to include a backup of critical data structures. This means that FAT32 volumes are less susceptible to a single point of failure than FAT16 volumes.
This is true if you are using a FAT32 enhanced driver and FAT32 enhanced fsck which should be win98 scandisk equivalent. however, the vfat support in linux and the corresponding fsck.vfat are not that advanced. vfat support has improved in that it can handle FAT32 lfn extensions, but fsck.vfat is still approximately equivalent to DOS6.0 chkdsk/scandisk. If you ever ran a DOS6.0 scandisk on a FAT32 filesystem (not a DOS 6.2 version which has been enhanced), then you will know what I mean.

as for sync/async mode, in SharpROM, I found that using sync mode on larger SD cards, ie 4GB cards, can prevent it from loosing its partition table, especially if the card is relatively full. When the card is empty, there is not much difference...
On OpenZaurus, sync mode causes the SD card to be extremely slow and almost unusable.
In pdaXrom depending on the kernel version and sd module being used, the default setting which I think is async, works fine...

[speculatrix]

so, maybe you need to run bochs or dosbox and fsck your fat32 flash memory card using that!

[daniel3000]

This is true if you are using a FAT32 enhanced driver and FAT32 enhanced fsck which should be win98 scandisk equivalent. however, the vfat support in linux and the corresponding fsck.vfat are not that advanced. vfat support has improved in that it can handle FAT32 lfn extensions, but fsck.vfat is still approximately equivalent to DOS6.0 chkdsk/scandisk. If you ever ran a DOS6.0 scandisk on a FAT32 filesystem (not a DOS 6.2 version which has been enhanced), then you will know what I mean.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157435\"][{POST_SNAPBACK}][/a][/div]

Aha, so you mean that fsck.vfat can much better recover FAT16 than FAT32 corruption?
Would you recommend FAT16 over FAT32 on pdaXrom for an SD card which mainly holds user files?


I just did some tests with sync mounting:

1GB SD card with FAT32 sync-mounted.

Then:
touch /mnt/card/test
ls /mnt/card --> test is shown. Wait a few seconds, then:
Removed the card without cleanly unmounting it.
inserted the card
ls /mnt/card --> test is NOT shown.

touch /mnt/card/test && sync. Don't wait, but:
Removed the card without cleanly unmounting it.
inserted the card
ls /mnt/card --> test IS shown.

So - does the sync option have no effect??
Only when waiting quite a long time after the touch command (maybe half a minute) before ejecting the card preserves the new file entry.

So maybe a cronjob which executes "sync" every second would be good  

But at least I get no "filesystem panic" messages from the kernel now.
When doing similar tests without -o sync, this happened often.

Maybe, FAT16 with -o sync is really the safest option.

daniel

[InSearchOf]

This is true if you are using a FAT32 enhanced driver and FAT32 enhanced fsck which should be win98 scandisk equivalent. however, the vfat support in linux and the corresponding fsck.vfat are not that advanced. vfat support has improved in that it can handle FAT32 lfn extensions, but fsck.vfat is still approximately equivalent to DOS6.0 chkdsk/scandisk. If you ever ran a DOS6.0 scandisk on a FAT32 filesystem (not a DOS 6.2 version which has been enhanced), then you will know what I mean.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157435\"][{POST_SNAPBACK}][/a][/div]

Aha, so you mean that fsck.vfat can much better recover FAT16 than FAT32 corruption?
Would you recommend FAT16 over FAT32 on pdaXrom for an SD card which mainly holds user files?


I just did some tests with sync mounting:

1GB SD card with FAT32 sync-mounted.

Then:
touch /mnt/card/test
ls /mnt/card --> test is shown. Wait a few seconds, then:
Removed the card without cleanly unmounting it.
inserted the card
ls /mnt/card --> test is NOT shown.

touch /mnt/card/test && sync. Don't wait, but:
Removed the card without cleanly unmounting it.
inserted the card
ls /mnt/card --> test IS shown.

So - does the sync option have no effect??
Only when waiting quite a long time after the touch command (maybe half a minute) before ejecting the card preserves the new file entry.

So maybe a cronjob which executes "sync" every second would be good  

But at least I get no "filesystem panic" messages from the kernel now.
When doing similar tests without -o sync, this happened often.

Maybe, FAT16 with -o sync is really the safest option.

daniel
[div align=\"right\"][a href=\"index.php?act=findpost&pid=157442\"][{POST_SNAPBACK}][/a][/div]

How about add the sync option to fstab

Late

[daniel3000]

How about add the sync option to fstab

[div align=\"right\"][a href=\"index.php?act=findpost&pid=157444\"][{POST_SNAPBACK}][/a][/div]

Should this make a difference?
I have added it to the /etc/sdcontrol script, and the card IS mounted with the sync option ("mount" shows (rw,sync) as the mount options).

daniel

[InSearchOf]

yea your right... I forgot sdcontrol mounts it... unless you umount and remount from the console... that uses fstab data...

Late