Help - Search - Members - Calendar
Full Version: Swap 32gb Cf For Microdrive
OESF Forums > Distros, Development, and Model Specific Forums > Distro Support and Discussion > Debian
ArchiMark
Decided to take the plunge and got a new 32GB A-Data CF Card today....

rolleyes.gif

Want to do this swap the right way and hopefully easy way....

I assume that I can 'clone' my current Debian EABI setup on my existing 6GB microdrive over to the new 32GB CF card, right?

Would this be the right way to do it?

1) Format new 32GB card and make 2 partitions; ext3 (/dev/hdc1 /mnt/cf) and a small swap (512MB on /dev/hdc2 /mnt/cf2).

2) Copy over current microdrive debian contents on hda2 over to new ext3 partition on 32GB CF card by doing:

CODE
#dd if=/dev/hda2 of=/dev/cf


* Note: hda1 on my existing microdrive is my swap partition.

So, is this correct way???

Thanks!

Mark
adf
QUOTE(ArchiMark @ Apr 1 2008, 07:20 PM) *
Decided to take the plunge and got a new 32GB A-Data CF Card today....

rolleyes.gif

Want to do this swap the right way and hopefully easy way....

I assume that I can 'clone' my current Debian EABI setup on my existing 6GB microdrive over to the new 32GB CF card, right?

Would this be the right way to do it?

1) Format new 32GB card and make 2 partitions; ext3 (/dev/hdc1 /mnt/cf) and a small swap (512MB on /dev/hdc2 /mnt/cf2).

2) Copy over current microdrive debian contents on hda2 over to new ext3 partition on 32GB CF card by doing:

CODE
#dd if=/dev/hda2 of=/dev/cf


* Note: hda1 on my existing microdrive is my swap partition.

So, is this correct way???

Thanks!

Mark


Oooooo Nooooo. How fast is it? Was it priced reasonably?
I've never moved stuff with dd like that--I'd have tried tar, but only on fa,iliarity, not on the belief that tar would be a better way to do it. You are probably right.
tux
cool.gif @Archimark: I think that your cloning progedure needs some input from someone who has tried it. Possibly ZDevil or some of the others who have tarred up filesystems might be able to give some good advice here about using their methods. smile.gif

@adf: Archimark is talking about using and A-Data card. I remember you posting about major problems getting Debian to work on your transplanted A-data card. Did you solve those problems?

Cheers
ArchiMark
QUOTE(adf @ Apr 2 2008, 12:06 AM) *
Oooooo Nooooo. How fast is it? Was it priced reasonably?
I've never moved stuff with dd like that--I'd have tried tar, but only on fa,iliarity, not on the belief that tar would be a better way to do it. You are probably right.


I'm not sure how fast it is....but it was priced very reasonably I though from SuperMediaStore.com....

$149.99 for the 32GB CF and $69.99 for a 16GB SDHC Card....

tongue.gif

QUOTE(tux @ Apr 2 2008, 01:05 AM) *
cool.gif @Archimark: I think that your cloning progedure needs some input from someone who has tried it. Possibly ZDevil or some of the others who have tarred up filesystems might be able to give some good advice here about using their methods. smile.gif

Sounds like a good idea, tux.....

ZDevil, where are you???.... or anyone else....

wink.gif


Cheers

speculatrix
QUOTE(ArchiMark @ Apr 2 2008, 04:20 AM) *
Decided to take the plunge and got a new 32GB A-Data CF Card today....


no, you definitely don't want to use "dd" to copy a file system between different types of media unless you guarantee the number of blocks in the partition is the same or larger on the target.

anyway, "cat" is faster - it uses more buffering!

the best way to copy a filesys is like this:

cd /mnt/partition1
tar cf - | (cd /mnt/partition2 ; tar xf - )

tar will copy all files and devices, preserve the datestamps and protections etc.

if you were l33t, you could probably also do it with cpio but I'm too lazy to look it up.

so, just partition the CF appropriately. oh, do NOT use ext3, the journalling will kill the card faster.
ArchiMark
QUOTE(speculatrix @ Apr 2 2008, 02:44 PM) *
QUOTE(ArchiMark @ Apr 2 2008, 04:20 AM) *
Decided to take the plunge and got a new 32GB A-Data CF Card today....


no, you definitely don't want to use "dd" to copy a file system between different types of media unless you guarantee the number of blocks in the partition is the same or larger on the target.

anyway, "cat" is faster - it uses more buffering!

the best way to copy a filesys is like this:

cd /mnt/partition1
tar cf - | (cd /mnt/partition2 ; tar xf - )

tar will copy all files and devices, preserve the datestamps and protections etc.

if you were l33t, you could probably also do it with cpio but I'm too lazy to look it up.

so, just partition the CF appropriately. oh, do NOT use ext3, the journalling will kill the card faster.


Thanks for your help with this, speculatrix!!

Will report back after I try this out...

wink.gif

ZDevil
QUOTE(speculatrix @ Apr 2 2008, 11:44 PM) *
[snipped]
oh, do NOT use ext3, the journalling will kill the card faster.

This may not be true in practice, as argued in this article, which in a way also applies to CF disks: http://wiki.eeeuser.com/ssd_write_limit
And given that my Z with 16GB vs. my Eee PC with 4GB ...
ArchiMark
QUOTE(ZDevil @ Apr 3 2008, 12:10 PM) *
QUOTE(speculatrix @ Apr 2 2008, 11:44 PM) *
[snipped]
oh, do NOT use ext3, the journalling will kill the card faster.

This may not be true in practice, as argued in this article, which in a way also applies to CF disks: http://wiki.eeeuser.com/ssd_write_limit
And given that my Z with 16GB vs. my Eee PC with 4GB ...


Thanks for that info/link....sounds reassuring as I have a feeling I won't be using my Z in 25 years....

wink.gif
adf
QUOTE
@adf: Archimark is talking about using and A-Data card. I remember you posting about major problems getting Debian to work on your transplanted A-data card. Did you solve those problems?


No, I went to pdaxii13. I am hoping the sharpbootloader method will work, though.
ArchiMark
QUOTE(speculatrix @ Apr 2 2008, 02:44 PM) *
anyway, "cat" is faster - it uses more buffering!


I just realized that I don't see 'cat' below in your instructions???

Also, I thought that 'cat' is for just viewing the content of a file?


QUOTE
the best way to copy a filesys is like this:
CODE
cd /mnt/partition1
tar cf - | (cd /mnt/partition2; tar xf - )


tar will copy all files and devices, preserve the datestamps and protections etc.


So, just want to make sure I understand this correctly for my situation of copying my hda2 partition with Debian EABI (hda1 is my swap partition) on it over to new partition on 32GB CF card....

So, here's the existing and new setups:

Existing 6GB Microdrive
hda1 is Swap Partition
hda2 is Debian EABI Partition

New 32GB CF Card
cf1 will be new Debian EABI Partition
cf2 will be new Swap Partition

CODE
cd /mnt/cf1
tar /mnt/hda2 - | (cd /mnt/cf1; tar xf - )


Does this look right?

Thanks!
iwantprogress
rsync also works and is faster if you need to redo.
speculatrix
QUOTE(ArchiMark @ Apr 4 2008, 07:35 PM) *
QUOTE(speculatrix @ Apr 2 2008, 02:44 PM) *
anyway, "cat" is faster - it uses more buffering!


I just realized that I don't see 'cat' below in your instructions???

Also, I thought that 'cat' is for just viewing the content of a file?


no, you could do, for example, "cat /dev/hda > /dev/hdb" to clone an entire hard drive

QUOTE
So, here's the existing and new setups:

Existing 6GB Microdrive
hda1 is Swap Partition
hda2 is Debian EABI Partition

New 32GB CF Card
cf1 will be new Debian EABI Partition
cf2 will be new Swap Partition

CODE
cd /mnt/cf1
tar /mnt/hda2 - | (cd /mnt/cf1; tar xf - )


Does this look right?

Thanks!


you're missing a dot

cd /mnt/hda2
tar cf - . | (cd /mnt/cf1 ; tar xf - )


you cd into the source directory so that when you do the tar the current directory is the "root" of all the files; the "-" in the tar command outputs the stream to stdout; the stream is piped into a separate shell (the brackets) which starts in the destionation directory and unpacks the incoming stream from stdin.

the same trick works over an ssh session.
ArchiMark
QUOTE(speculatrix @ Apr 4 2008, 02:58 PM) *
QUOTE(ArchiMark @ Apr 4 2008, 07:35 PM) *
QUOTE(speculatrix @ Apr 2 2008, 02:44 PM) *
anyway, "cat" is faster - it uses more buffering!


I just realized that I don't see 'cat' below in your instructions???

Also, I thought that 'cat' is for just viewing the content of a file?


no, you could do, for example, "cat /dev/hda > /dev/hdb" to clone an entire hard drive

QUOTE
So, here's the existing and new setups:

Existing 6GB Microdrive
hda1 is Swap Partition
hda2 is Debian EABI Partition

New 32GB CF Card
cf1 will be new Debian EABI Partition
cf2 will be new Swap Partition

CODE
cd /mnt/cf1
tar /mnt/hda2 - | (cd /mnt/cf1; tar xf - )


Does this look right?

Thanks!


you're missing a dot

cd /mnt/hda2
tar cf - . | (cd /mnt/cf1 ; tar xf - )


you cd into the source directory so that when you do the tar the current directory is the "root" of all the files; the "-" in the tar command outputs the stream to stdout; the stream is piped into a separate shell (the brackets) which starts in the destionation directory and unpacks the incoming stream from stdin.

the same trick works over an ssh session.


THANKS for the explanation/clarification, speculatrix!!

Think I got it now....

smile.gif
ArchiMark
QUOTE(iwantprogress @ Apr 4 2008, 01:59 PM) *
rsync also works and is faster if you need to redo.


Thanks for your suggestion too, iwantprogress, didn't know about rsync....

Just googled it and found the rsync website....sounds interesting....

wink.gif

ArchiMark
OK, tried the 'cat' approach and here's what I get:

CODE
#cat /dev/hda2 > /dev/hdc1
#df
Filesystem   1K-blocks   Used                    Available  Used%      Mounted   on
/dev/hda2      5589680       1317484        3988248  25%      /
tmpfs      31180           0        31180    0%      /lib/init/rw
udev      10240         80        10160    1%      /dev
tmpfs      31180           4        31176    1%      /dev/shm
/dev/hdc1      5184312     -25616124    30516488     -     /mnt/cf1

#cd /mnt/cf1
EXT3-fs error (device hdc1): ext_readdir: bad entry in directory #2: directory
entry across blocks - offset=0, inode=4294967295, rec_len=65536, name_len=255


So, should I just try the other way or did I do something wrong???

blink.gif

Thanks!
rogalian
Cat gives same results as dd - won't work unless devices are the same or close enough

Use the tar method!
speculatrix
given that the cards are different sizes and therefore your partitioning needs to be different, you're going to run into a world of pain copying the card raw block style; just use "fdisk" to partition the target, "mkfs.ext2 /dev/X" to format it, then mount and copy files. really, it's actually not hard :-)
ArchiMark
QUOTE(speculatrix @ Apr 4 2008, 02:58 PM) *
you're missing a dot

cd /mnt/hda2
tar cf - . | (cd /mnt/cf1 ; tar xf - )


you cd into the source directory so that when you do the tar the current directory is the "root" of all the files; the "-" in the tar command outputs the stream to stdout; the stream is piped into a separate shell (the brackets) which starts in the destionation directory and unpacks the incoming stream from stdin.

the same trick works over an ssh session.


Just to clarify, in looking at the above, shouldn't the first 'tar' above be for the 'hda2' partition, not 'cf' as shown??

In other words, shouldn't the above look like this?

CODE
#cd /
#tar / - . | (cd /mnt/cf1; tar xf - )


Thanks!
ArchiMark
QUOTE(speculatrix @ Apr 7 2008, 08:32 AM) *
given that the cards are different sizes and therefore your partitioning needs to be different, you're going to run into a world of pain copying the card raw block style; just use "fdisk" to partition the target, "mkfs.ext2 /dev/X" to format it, then mount and copy files. really, it's actually not hard :-)



I did format partition after I created it, speculatrix....so that's already done...

As for copying files, I can do that, just assumed that this might not work as when you try to do that with a Windoze install...it won't make for a bootable drive, you have to use a cloning app in order to get an exact duplicate of drive that boots....

So, if Linux is different than Windoze in this regard, then no problem, I'll just copy the directories over to the CF card....


Hope this makes sense....

speculatrix
"tar cf" is the command, "c" means create, "f" means the next argument is the output file (- means use stdout). "tar xf" means extract with source being stdin.

if you're copying the root partition, you will need to exclude the target mount point, eg.

cd /
tar cf - . --exclude=/mnt/newdir | ( cd /mnt/newdir ; tar xf - )
speculatrix
p.s. if you're copying with tar, the target disk/partition needs to be formatted and mounted in advance.

if you're cloning with cat or dd, the partition should NOT be mounted at the time.
ArchiMark
QUOTE(speculatrix @ Apr 7 2008, 09:10 AM) *
"tar cf" is the command, "c" means create, "f" means the next argument is the output file (- means use stdout). "tar xf" means extract with source being stdin.

if you're copying the root partition, you will need to exclude the target mount point, eg.

cd /
tar cf - . --exclude=/mnt/newdir | ( cd /mnt/newdir ; tar xf - )


I get it now, speculatrix!!!....I thought that the 'cf' there was referencing 'cf' as in 'compact flash'......

rolleyes.gif


QUOTE(speculatrix @ Apr 7 2008, 09:10 AM) *
p.s. if you're copying with tar, the target disk/partition needs to be formatted and mounted in advance.

if you're cloning with cat or dd, the partition should NOT be mounted at the time.


Thanks for clarifying that....so, I did the clone with 'cat' wrong as I had mounted the partition I think....

Meanwhile, I'm just copying over directories to CF card now....

ArchiMark
Copying over directories...most copying over OK, but have had a few error messages:


CODE
#cp -r /proc /mnt/cf1
Warning: /proc/ide/hd?/settings interface is obsolete, and will be removed soon!
cp: cannot open `/proc/sys/net/ipv4/route/flush' for reading: Permission denied
cp: cannot open `/proc/sys/net/ipv6/route/flush' for reading: Permission denied
process `cp' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time:
Use net.ipv6.neigh.retrans_time_ms instead.
cp: reading `/proc/sysrg-trigger': Input/output error


No more HD green light activity showing and command prompt has not returned after 10 minutes...

????

blink.gif

Thanks!
rogalian
The stuff in /proc you needn't worry about, it's auto generated by the kernel
speculatrix
ooops, yes, exclude /proc and /sys too. probably not a bad idea to clean out /tmp too.
ArchiMark
Thanks for the advice, rogalian & speculatrix!

smile.gif

speculatrix
let us know when you're back up and running.
ArchiMark
QUOTE(speculatrix @ Apr 8 2008, 01:52 PM) *
let us know when you're back up and running.


Thanks speculatrix.....

Meanwhile, copied over directories, cleaned out 'tmp'.....

Just did 'df' and here's what I see:
CODE
Filesystem     1K-blocks      Used     Available    Use%     Mounted on
/dev/hda2     5589680      1317404     3988328     25%    /
tmpfs               31180                 0        31180       0%    /lib/init/rw
udev                10240               80        10160       1%    /dev
tmpfs               31180                 4        31176       1%    /dev/shm
/dev/hdc1     5589680      2419708     2886024     46%    /mnt/cf1


So......why does my existing microdrive with Debian EABI only use 25% of the drive and the new 32GB CF card use 46% of drive???

Maybe I'm just being overly cautious with all of this...but I'm trying to avoid situation where I yank out microdrive, take apart my Z and install CF card, put Z back together, and voila! it doesn't boot up or something like that....

Thanks!




ajnk
I had to do something similar last night, when I managed to make my 16gb CF unbootable, fortunately i had my 4gb I was using until about 1 month ago :-

1) Attached 4gb microdrive to laptop using USB adapter, checked the filesystem (just to be safe).
2) Mount 4gb microdrive (/media/disk in this case)
3) cd /media/disk
4) copy contents to laptop HD (I wanted a) a backup cool.gif a fast way to redo the process if I messed up)
mkdir /home/andy/z/cf_recovery
find . | cpio -pdmv /home/andy/z/cf_recovery
5) wait .......
6) once the copy was complete umount /media/disk
7) Disassemble Z (c3100 in my case)
9) Retrieve CF card from Z, attach to lappy via USB.
10) It automounted on /media/disk, so I unmounted it.
11) Create an ext2 filesystem (yes I want ext3 journaling, but i am going to dump a lot of data in a stable environment so I don't need the journal - it would only slow things down).
mke2fs /dev/hdb1
tune2fs -i 0 -c 0 /dev/hda1
(turns off fsck after a number of mounts).
12) mount the cf card (mount /dev/hdb1 /media/disk)
13) cd /home/andy/z/cf_recovery
13) find . | cpio -pdmv /media/disk
14) wait .....
15) umount cf card, insert in Z, rebuild Z - hey presto - working fs

BTW. If you need to repeat step 13) for any reason either remake the filesystem or add the 'u' flag (unconditional overwrite) to the cpio.

Hope this helps
ajnk

(Then I did an apt-get update ; apt-get upgrade )
ZDevil
Good to know that things are working now.
ajnk: so do you mean the latest yonggun kernel can boot off an ext2 partition? I remember the previous one only takes ext3 ...
ajnk
QUOTE(ZDevil @ Apr 9 2008, 12:19 PM) *
Good to know that things are working now.
ajnk: so do you mean the latest yonggun kernel can boot off an ext2 partition? I remember the previous one only takes ext3 ...



I don't know the answer to your question as I enabled the journal (tune2fs -j /dev/hda1) before inserting the CF into the Z. I left journaling off while copying about 2.5Gb of data to it (the recovery from the laptop) - it sped things up A LOT and stopped my laptop from crashing. I almost looked like the CF was thrashing - but it's memory not a disk ?!?!

Also edited /etc/fstab on the Z to add the noatime option to the filesystem mount - to turn off recording off access times - again should help performance.
speculatrix
QUOTE(ArchiMark @ Apr 9 2008, 04:18 AM) *
So......why does my existing microdrive with Debian EABI only use 25% of the drive and the new 32GB CF card use 46% of drive???


hmm, odd.

try doing an "ls -laR" within each, redirecting output to file and diff'ing them? might need to "grep -v ^d" to omit directory names.
ArchiMark
QUOTE(speculatrix @ Apr 9 2008, 09:42 AM) *
QUOTE(ArchiMark @ Apr 9 2008, 04:18 AM) *
So......why does my existing microdrive with Debian EABI only use 25% of the drive and the new 32GB CF card use 46% of drive???


hmm, odd.

try doing an "ls -laR" within each, redirecting output to file and diff'ing them? might need to "grep -v ^d" to omit directory names.


Thanks for the suggestion, speculatrix....

Meanwhile, decided to just bite the bullet and deleted everything on CF card and copied over stuff again....

Now, when I did 'df' I get:

CODE
#df
Filesystem    1K-blocks     Used    Available    Use%  Mounted on
/dev/hda2     5589680     1317392    3988340    25%   /
tmpfs    31180       0       31180    0%   /lib/init/rw
udev    10240    80       10160    1%   /dev
tmpfs    31180    0       31180    1%  /dev/shm
/dev/hdc1     5589680     1386936    3918796    27%  /mnt/cf1


So, hda2 and cf1 are close now, but still slightly different.

Is this OK?? Or ???


Also, since I've:

1) 'flipped' location of Debian from second partition (hda2) on microdrive to first partition (cf1) on 32GB CF card and

2) just copied over directories as-is from microdrive to CF card

Is there going to be any issue when I yank out microdrive and put in CF card in it's place?

In other words is there any other things I need to do in order to make sure that CF will be recognized and then boot up Debian after CF card is installed?

Thanks!
speculatrix
you will find that a file copy using tar, rsync or cpio will be different.. say, for example, a directory had loads of files in it and the directory occupied lots of blocks, but then most of the files were deleted, the directory's usage wouldn't diminish. back to the original size.

when you mkfs, you can specify how much of the disk should be reserved - the default IIRC is 10% but on large disks that's often more than necesssary. On a 100GB+ disk I'll drop it to 5%, on a 750GB+ I'll use just 3. DONT set to 0 because it'll cause fragmentation as space runs out. You can change the % with tune2fs.

--edit--
if you're moving debian from one part'n to another you may need to flash a different kernel, as the location of the root part'n is wired in at build time. See the multi-boot thread on debian for details of the different kernels to be used for different root locations.
ArchiMark
QUOTE(speculatrix @ Apr 11 2008, 01:54 PM) *
you will find that a file copy using tar, rsync or cpio will be different.. say, for example, a directory had loads of files in it and the directory occupied lots of blocks, but then most of the files were deleted, the directory's usage wouldn't diminish. back to the original size.

when you mkfs, you can specify how much of the disk should be reserved - the default IIRC is 10% but on large disks that's often more than necesssary. On a 100GB+ disk I'll drop it to 5%, on a 750GB+ I'll use just 3. DONT set to 0 because it'll cause fragmentation as space runs out. You can change the % with tune2fs.

--edit--
if you're moving debian from one part'n to another you may need to flash a different kernel, as the location of the root part'n is wired in at build time. See the multi-boot thread on debian for details of the different kernels to be used for different root locations.


Thanks for the explanations, speculatrix!

So, I guess that I can

1) swap out internal drive, then
2) reflash kernel from SD card, then
3) let 3200 boot up to new CF card....

and hope this Z boots up properly....

smile.gif
maemorandum
Does it make sense replacing the internal microdrive with a bigger compact-flashcard? So, if you use a part of your cf as swap-image in Debian it will probably have a lot of writing cycles on it. I was told, a cf-card has not as many overwriting cycles (about 200.000 or so) as a microdrive blink.gif . Anyway, writing a cf-card is much slower than reading it - and might be slower than reading a microdrive.

Pro: Bigger, less power consumption laugh.gif
Contra: Might be slower, shorter life, when used as swap

agree?
ArchiMark
QUOTE(maemorandum @ Apr 11 2008, 03:52 PM) *
Does it make sense replacing the internal microdrive with a bigger compact-flashcard? So, if you use a part of your cf as swap-image in Debian it will probably have a lot of writing cycles on it. I was told, a cf-card has not as many overwriting cycles (about 200.000 or so) as a microdrive blink.gif . Anyway, writing a cf-card is much slower than reading it - and might be slower than reading a microdrive.

Pro: Bigger, less power consumption laugh.gif
Contra: Might be slower, shorter life, when used as swap

agree?


Good points, maemorandum....

However, please see earlier posts in this thread, like ZDevil's post #7 that has link to webpage on this subject, it appears that it might take 25 years for the effects that you refer to show up...


speculatrix
so, archimark, have you done it yet? we eager readers are desperate to know!
ArchiMark
QUOTE(speculatrix @ Apr 12 2008, 12:34 PM) *
so, archimark, have you done it yet? we eager readers are desperate to know!


Not yet...

Per my other thread on XFCE4, stylus stopped working, no response to screen taps, want to fix that first then, do swap out.....so that I know I have fully working setup when I swap...

So, if I can fix this today, then I'll swap tonight or if not, tomorrow....will report back after I do it...

Thanks.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2014 Invision Power Services, Inc.