OESF | ELSI | pdaXrom | OpenZaurus | Zaurus Themes | Community Links | Ibiblio

IPB

Welcome Guest ( Log In | Register )

> Static Site Snapshot/mirror, for storing on the Z
speculatrix
post Nov 10 2006, 02:46 AM
Post #1





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



Just as you can get a snapshot of wikipedia and store on your PC (or even PDA), it occurred to me that a snapshot of the OESF forum would be a very useful thing if combined with free text search.

There's a huge amount of wisdom on it (a lot of which should be in the wiki, but isn't), and so an archive would be a real asset. The low-graphics version of course would be best, but provided the archive was created without the attachments, it'd be OK as there'd be only one copy of the site.

I did try a speed-throttled wget once, but it wasn't too satisfactory. huh.gif

Any chance of considering being able to do this on the server itself and make a monthly snapshot downloadable in .zip or .tar.bz2? Some of us could burn CDs or DVDs to send to people without broad-band.

thanks
Paul
Go to the top of the page
 
+Quote Post
 
Start new topic
Replies
ShiroiKuma
post Nov 15 2006, 02:33 AM
Post #2





Group: Members
Posts: 902
Joined: 22-May 04
Member No.: 3,385



I think it's not a very smart idea to download forums with a tool lik Htttrack. This forum is based on PHP scripts which interface to a database. Downloading with htttrack makes the forum engine generate thousands of pages, resulting in high usage and high traffic.

It should not be too hard to make the database available for download regularly in a compressed format. Then one would create scripts to serve the database locally on the Z...
Go to the top of the page
 
+Quote Post
speculatrix
post Nov 15 2006, 04:27 AM
Post #3





Group: Admin
Posts: 3,281
Joined: 29-July 04
From: Cambridge, England
Member No.: 4,149



QUOTE(ShiroiKuma @ Nov 15 2006, 11:33 AM)
I think it's not a very smart idea to download forums with a tool lik Htttrack. This forum is based on PHP scripts which interface to a database. Downloading with htttrack makes the forum engine generate thousands of pages, resulting in high usage and high traffic.
*


httrack has nice features to limit the load it puts on servers, and in fact the latest version has a restriction of 100kbps bandwidth use.

I made sure it restricted the number of parallel page fetches too, so that it too quite a while to download.

I've got building work on at home and they've cut the power again, so I can't access my fileserver and do the upload at the moment.
Go to the top of the page
 
+Quote Post

Posts in this topic
speculatrix   Static Site Snapshot/mirror   Nov 10 2006, 02:46 AM
daniel3000   This is a terrific idea and I also wish this would...   Nov 10 2006, 03:42 AM
speculatrix   QUOTE(daniel3000 @ Nov 10 2006, 12:42 PM)This...   Nov 14 2006, 01:33 PM
desertrat   QUOTE(speculatrix @ Nov 14 2006, 09:33 PM)The...   Nov 14 2006, 05:22 PM
speculatrix   well, the first run seems to be complete... create...   Nov 14 2006, 03:30 PM
Jon_J   speculatrix, I tried using the program you listed ...   Nov 14 2006, 08:39 PM
speculatrix   QUOTE(Jon_J @ Nov 15 2006, 05:39 AM)speculatr...   Nov 15 2006, 02:21 AM
ShiroiKuma   I think it's not a very smart idea to download...   Nov 15 2006, 02:33 AM
speculatrix   QUOTE(ShiroiKuma @ Nov 15 2006, 11:33 AM)I th...   Nov 15 2006, 04:27 AM
speculatrix   I've just had a thought... what's the limi...   Nov 15 2006, 04:29 AM
ShiroiKuma   QUOTE(speculatrix @ Nov 15 2006, 01:29 PM)I...   Nov 15 2006, 04:38 AM
matthis   Actually these kind of forums store data in a sql ...   Nov 15 2006, 07:37 AM
zmiq2   Or maybe a link to a file with the daily activity,...   Nov 15 2006, 07:57 AM
speculatrix   hmm, well, it's a pretty damn big file when zi...   Nov 21 2006, 02:00 PM
speculatrix   The snapshot is now up as a mirror... see mainstre...   Nov 22 2006, 07:45 AM
speculatrix   Can I ask DZ and other admins to please consider a...   Feb 21 2007, 10:41 AM
Antikx   QUOTE(speculatrix @ Feb 21 2007, 12:41 PM)Can...   Feb 21 2007, 02:27 PM
speculatrix   QUOTE(Antikx @ Feb 21 2007, 11:27 PM)QUOTE(sp...   Feb 21 2007, 03:34 PM
Antikx   sorry for stating the obvious.   Feb 21 2007, 05:55 PM
speculatrix   OK, I'm running whtt again - 2 connections per...   Feb 22 2007, 12:28 PM
speculatrix   Ok, it completed after 8 hours, about 30,000 pages...   Feb 22 2007, 11:22 PM


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th July 2014 - 07:11 AM