OESF | ELSI | pdaXrom | OpenZaurus | Zaurus Themes | Community Links | Ibiblio

IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
> Static Site Snapshot/mirror, for storing on the Z
speculatrix
post Nov 10 2006, 02:46 AM
Post #1





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



Just as you can get a snapshot of wikipedia and store on your PC (or even PDA), it occurred to me that a snapshot of the OESF forum would be a very useful thing if combined with free text search.

There's a huge amount of wisdom on it (a lot of which should be in the wiki, but isn't), and so an archive would be a real asset. The low-graphics version of course would be best, but provided the archive was created without the attachments, it'd be OK as there'd be only one copy of the site.

I did try a speed-throttled wget once, but it wasn't too satisfactory. huh.gif

Any chance of considering being able to do this on the server itself and make a monthly snapshot downloadable in .zip or .tar.bz2? Some of us could burn CDs or DVDs to send to people without broad-band.

thanks
Paul
Go to the top of the page
 
+Quote Post
daniel3000
post Nov 10 2006, 03:42 AM
Post #2





Group: Members
Posts: 1,002
Joined: 28-April 05
From: Germany
Member No.: 7,027



This is a terrific idea and I also wish this would be possible.
I think a forum like this is based on a database. it should not be too hard to dump the database content into pute text files (or HTML maybe).

If it isn't possible on the server side, there is still the "lo-fi" versoin of the forums which are simpler HTML pages and probalby easier and faster to collect via a tool like wget or plucker. So this may be another idea.

Another nice feature would be to install an NNTP gateway or POP/IMAP server inorder to make it possible to download the forum contents to client programs like email clients or usenet clients. If it would be possible to add content (reply, add topics) also from these clients, this would be even better.

But this would probablyl be a major effort and since we can be thankful that some nice people drive this forum without profit, I totally understand if this will never happen :-)

daniel
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 14 2006, 01:33 PM
Post #3





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



QUOTE(daniel3000 @ Nov 10 2006, 12:42 PM)
This is a terrific idea and I also wish this would be possible.
I think a forum like this is based on a database. it should not be too hard to dump the database content into pute text files (or HTML maybe).

If it isn't possible on the server side, there is still the "lo-fi" versoin of the forums which are simpler HTML pages and probalby easier and faster to collect via a tool like wget or plucker. So this may be another idea.


I now found a program which will do such a job, and am trying it out... the good thing is that it allows me to choose to only download URLs with "lofiversion" in them and to throttle bandwidth. The problem with wget is that it doesn't fix links, whereas this program does.

http://www.httrack.com/

When it's finished, I will see what the results are like and offer a download on my website.

QUOTE
Another nice feature would be to install an NNTP gateway or POP/IMAP server inorder to make it possible to download the forum contents to client programs like email clients or usenet clients. If it would be possible to add content (reply, add topics) also from these clients, this would be even better.

But this would probablyl be a major effort and since we can be thankful that some nice people drive this forum without profit, I totally understand if this will never happen :-)

daniel
*


I did ask about an RSS feed, but I think this forum doesn't have the facility.

Paul
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 14 2006, 03:30 PM
Post #4





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



well, the first run seems to be complete... created 8600-ish HTML files... tweaking the results now, such as putting in the CSS files and such. seems to work quite well.
Go to the top of the page
 
+Quote Post
desertrat
post Nov 14 2006, 05:22 PM
Post #5





Group: Members
Posts: 742
Joined: 15-October 05
From: Gulag, Siberia
Member No.: 8,322



QUOTE(speculatrix @ Nov 14 2006, 09:33 PM)
The problem with wget is that it doesn't fix links, whereas this program does.

Doesn't the --convert-links, --html-extension options cover it?
Go to the top of the page
 
+Quote Post
Jon_J
post Nov 14 2006, 08:39 PM
Post #6





Group: Members
Posts: 1,843
Joined: 31-December 05
From: Illinois USA
Member No.: 8,821



speculatrix,
I tried using the program you listed to make an offline copy of this wiki site:
http://www.uesp.net/wiki/Oblivion:Oblivion

It's a wiki about a computer RPG game.
"The Elder Scrolls IV - Oblivion"

It downloaded about 9,000 pages that resulted in a 120MB.
It took 1 hour on my adsl connection.
This is just for a computer game...!?
Is there any way to know beforehand, how much data is going to be downloaded?
I was thinking about placing this wiki on my Zaurus, but it ended up being way too big for just one game.
I decided to cancel this because it was downloading 8 other previous versions of (Elder Scrolls game wikis) which are in the left-hand frame.

I had been just saving individual web pages that interested me from this wiki previously.
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 15 2006, 02:21 AM
Post #7





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



QUOTE(Jon_J @ Nov 15 2006, 05:39 AM)
speculatrix,
I tried using the program you listed to make an offline copy of this wiki site:
http://www.uesp.net/wiki/Oblivion:Oblivion
It downloaded about 9,000 pages that resulted in a 120MB.
It took 1 hour on my adsl connection.
*


make sure you only include HTML files and CSS, don't allow it to leave the domain, ensure you specify the correct url to match just the wiki, and so on. there are countless options!

QUOTE(desertrat)
Doesn't the --convert-links, --html-extension options cover it?


Hmm, possibly, but I was struggling to get it to convert the query string and fix everything and stay on the lofiversion URL. This windows program just made it easier to do.

Paul
Go to the top of the page
 
+Quote Post
ShiroiKuma
post Nov 15 2006, 02:33 AM
Post #8





Group: Members
Posts: 902
Joined: 22-May 04
Member No.: 3,385



I think it's not a very smart idea to download forums with a tool lik Htttrack. This forum is based on PHP scripts which interface to a database. Downloading with htttrack makes the forum engine generate thousands of pages, resulting in high usage and high traffic.

It should not be too hard to make the database available for download regularly in a compressed format. Then one would create scripts to serve the database locally on the Z...
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 15 2006, 04:27 AM
Post #9





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



QUOTE(ShiroiKuma @ Nov 15 2006, 11:33 AM)
I think it's not a very smart idea to download forums with a tool lik Htttrack. This forum is based on PHP scripts which interface to a database. Downloading with htttrack makes the forum engine generate thousands of pages, resulting in high usage and high traffic.
*


httrack has nice features to limit the load it puts on servers, and in fact the latest version has a restriction of 100kbps bandwidth use.

I made sure it restricted the number of parallel page fetches too, so that it too quite a while to download.

I've got building work on at home and they've cut the power again, so I can't access my fileserver and do the upload at the moment.
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 15 2006, 04:29 AM
Post #10





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



I've just had a thought... what's the limit on the number of files in the same directory on a fat32 memory card? Would the forum archive actually be storable on flash?
Go to the top of the page
 
+Quote Post
ShiroiKuma
post Nov 15 2006, 04:38 AM
Post #11





Group: Members
Posts: 902
Joined: 22-May 04
Member No.: 3,385



QUOTE(speculatrix @ Nov 15 2006, 01:29 PM)
I've just had a thought... what's the limit on the number of files in the same directory on a fat32 memory card? Would the forum archive actually be storable on flash?
*

I don't think this should be a problem really, since you probably should put it into a squashfs archive anyway and mount it on the Z. That way you'll save on space bigtime, because this is mostly text.
Go to the top of the page
 
+Quote Post
matthis
post Nov 15 2006, 07:37 AM
Post #12





Group: Members
Posts: 218
Joined: 12-February 06
From: Tokyo
Member No.: 9,124



Actually these kind of forums store data in a sql database. The admistrator has a button to dump/save all the messages, so asking for a dump could be a good start.
Go to the top of the page
 
+Quote Post
zmiq2
post Nov 15 2006, 07:57 AM
Post #13





Group: Members
Posts: 385
Joined: 3-December 03
Member No.: 1,038



Or maybe a link to a file with the daily activity, with all posts modified on that date, so you don't need to download the whole db!
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 21 2006, 02:00 PM
Post #14





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



hmm, well, it's a pretty damn big file when zipped up - over 60MB! It took a long time to spider the forum with the rate throttled right down - 21000 files or so!

I'm uploading the file now as a .zip to my website at http://www.zaurus.org.uk/downloads.html , because that way people who want a cramfs/squashfs mountable archive can create one, and people with windows can unpack it and use a local search function like google desktop to index it.

Paul
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 22 2006, 07:45 AM
Post #15





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



The snapshot is now up as a mirror... see mainstream discussion at
http://www.oesf.org/forums/index.php?showtopic=22041

for more details
Go to the top of the page
 
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st December 2014 - 04:06 AM