OESF Portables Forum
Everything Else => Embedded Linux Hardware / Software Index => Everything Else => Archived Forums => ELSI General Discussion => Topic started by: dz on September 23, 2004, 03:32:35 am
-
Ok, so I thought I'd keep you guys informed of the progress. I've begun the main bulk of the site. What's done so far is a preliminary layout and the login/registration system. I've decided to keep the accounts db different than that of the one we use here at the ZUG. What I did though is add a field in the 'options' page for the accounts where you can enter your ZUG username. What I plan to do with that is if someone wants to message you from there, they can enter a message and you have the option of it being automatically inserted here in the ZUG messageboard messages or having it be emailed to you. As well, for the project subscriptions, if there's an update, it can do the same.
I'm going to start production on the main core of the ZSI2 now, which is the actual projects themselves. I'll explain the database structure and the features so if any of you can think of anything else, those will be added.
A project has the following fields: id, name, unixname, screenshot, currentVersion, resolution, rom, model, processor, description, dateAdded, views, license, homepage, owner, and updated.
name: Name of the project.
unixname: Name of the projects filename. This will have no spaces, commas, ~'s; things like that.
currentVersion: I plan to have the ability to put multiple versions on the site and to have multiple branches. Those of you familiar with Sourceforge understand how this works. You can have a development branch and a stable branch, and have multiple versions of the program in there. The currentVersion field is set to whatever you consider the latest version to be.
Resolution: Admin will be able to add/edit/remove different resolutions. If Sharp decides to make another Zaurus and it has a resolution of 800x600 or something, we'll be able to add it easily.
Rom,Processor,License: Same as resolution where admin has complete control.
Model: I put this in so that if we do decide to take this to more than just the Zaurus, we can simply add the other models using the Admin page. Initially though, this will be something like Zaurus c860, Zaurus 5500, et al.
owner: This will be linked to your account.
updated: The last time the project has been updated.
If this sounds at all complicated, give me a few days and you'll see how simple it really is. Some of the stuff can be dropdown boxes (For example licenses), and some can be checkboxes (Resolution for projects that can function on multiple resolutions for example).
That is what is going to be programmed next. After that, the main chunk of the programming is done and I can add the 'features'.
Ok well, time to go to work. Let me know if anyone thinks of any new ideas
Roy
-
Eee, almost forgot the categories field. That one's self explanatory; I just forgot to mention it.
-
Will you have multiple categories, like killefiz did?
For example a command line app like nmap is in both 'command line' and 'network', etc.
The problem this had on killefiz (occasionally) was that people were either stupid (and set it to the wrong category) or were trying to get more exposure (slightly cynical I know) or perhaps just forgot and it defaulted.
Another quicky, will it be possible for anyone (registered) to edit the entries (even if they didn't create it), so that amendments/corrections can be made?
Si
P.S. What would also be nice is something like ipkgfind.handhelds.org... but I'm not sure how difficult it would be to implement (as the files will not be hosted locally).
-
Will you have multiple categories, like killefiz did?
Yes, you will have the ability to put your project into multiple categories.
Another quicky, will it be possible for anyone (registered) to edit the entries (even if they didn't create it), so that amendments/corrections can be made?
Not for anyone registered to; but I am adding the ability to have mods. If there is a problem, an entry can be flagged and mods can check it out.
The problem this had on killefiz (occasionally) was that people were either stupid (and set it to the wrong category) or were trying to get more exposure (slightly cynical I know) or perhaps just forgot and it defaulted.
Hopefully the mod system will help this. If an entry is clearly out of place, than it can be taken out of that area. Also, I'm debating on limiting you to 3 categories at the most.
-
P.S. What would also be nice is something like ipkgfind.handhelds.org... but I'm not sure how difficult it would be to implement (as the files will not be hosted locally).
actually, I think it was decided that we would host the files locally... (at least for the files we can get)
-
actually, I think it was decided that we would host the files locally... (at least for the files we can get)
That's good. In that case could an ipkgfind style function be added? That is:
search by package name
search by package description
search by file contained in package
These can all be done on all, or just a section of the files, etc.
I've no idea how it works for them, but I presume they host a db which catalogues all of the files. This would presumably not be too difficult to do automagically. No rush though, it would just be a nice feature.
Si
-
I think this would be a good idea: Dynamic feeds, Like those with accounts can create their own feeds with packages on the site, or create a feed with a package and all its dependancies in the same feed and nothing else.
-
Question: When a user enters a project, should they be allowed to select more than one of the following:
resolution
rom
model
processor
And if so, should I limit to how many?
-
Question: When a user enters a project, should they be allowed to select more than one of the following:
resolution
rom
model
processor
And if so, should I limit to how many?
Yes,
at least for ROM & model.
The processor and resolution might be more restrictive for compatibility.
Please try not to limit the number (if possible from database table layout / screen layout).
-- hns
-
Remember about paging - ZSI doesn't have it so browsing thru entries is a PITA.
Field which should exist is DISTRO - OZ 3.5.1 stuff won't work with older one etc.
-
It would be nice to have the dependencies listed (from the control file, with links to the files like ipkgfind.handhelds.org), so that one could download the whole lot in one go (manually rather than by creating an automated feed - though this would be nice too, but more work).
Si
-
Field which should exist is DISTRO - OZ 3.5.1 stuff won't work with older one etc.
Isn't that basically the same as ROM?
What might be good is fields for version of compiler (e.g. gcc3, gcc2.95) since they like to break compatibility and also even libc version. So for example, while there may be no developer/user comments saying an app works on your ROM, you would still have a clue as to whether it will work at all from these fields.
-
What might be good is fields for version of compiler (e.g. gcc3, gcc2.95) since they like to break compatibility and also even libc version. So for example, while there may be no developer/user comments saying an app works on your ROM, you would still have a clue as to whether it will work at all from these fields.
Compiler and libc version added.
Most of the fields are optional. Only ones I require are:
rom
model
processor
license
I'm going to keep compiler and libc version optional.
-
Most of the fields are optional. Only ones I require are:
rom
model
processor
license
Why is processor mandatory?
If the model name is required, the processor is fixed by that.
SL-A300 -> (?)
SL-5000 -> (?)
SL-5500 -> SA 1110 - 206 MHz
C860 -> XScale PXA255 - 400 MHz
SL6000 -> StrongARM - 400 MHz
iPAQ 3900 -> (?)
etc.
Probably, we also should have an "undefined" or "arbitrary"/"universal" value for 'model' and only then make the 'processor' a required field.
-- hns
-
I'd be tempted to get rid of the model name and just go for processor and screen size(s) which are the important factors here (e.g. all Cxxx machines are effectively the same). However this may confuse people...
Si
-
However this may confuse people...
I agree. Reading the model name you are developing on/for is much easier than identifying the processor. Or the new zsi could have a built-in translation table and make a default recommendation?
-- hns
-
I'd be tempted to get rid of the model name and just go for processor and screen size(s) which are the important factors here (e.g. all Cxxx machines are effectively the same). However this may confuse people...
Si
Ya, I suppose that makes sense.
K, model is now out of the picture. I will try and have something up for you guys to look at within the night. If not tonight, then give me a day or so.
On top of this, I have work and college, so I'm tryin to get as much done with as much time as I have.
-
However this may confuse people...
I agree. Reading the model name you are developing on/for is much easier than identifying the processor. Or the new zsi could have a built-in translation table and make a default recommendation?
-- hns
Ahh, I didnt see this post before.
Hmm.. perhaps, I could have an option for 'beginner' or 'advanced' mode.
Beginner mode doesn't show things like libc, compiler, processor, et al. It'll show basic things like Model and Rom.
Let me know what you think..
-
For the rom and processor choices, should they be limited to only one choice?
-
I've seen instances where something does not work for the 750, but works for the 760/860 and something work for the 750/760, but not the 860. They all have the same screen and CPU. I think model is a more accurate required description.
-
Before I go on, I'd like to see if we could come to one solution. Don't wanna have to reprogram a chunk of it :-/
-
Before I go on, I'd like to see if we could come to one solution.
From the point of view of a non linux geek, model number has to be the way to go, in conjunction with ROM image details - anyone can easily compare their Z with these two fields and work out whether or not the software will work. It also allows for nice searches - eg "show me all the software that will work with my Z"...
I'm happy for an advanced mode which shows details of lib dependencies, but that's really not going to appeal to a 'normal' user.
-
I've seen instances where something does not work for the 750, but works for the 760/860 and something work for the 750/760, but not the 860. They all have the same screen and CPU. I think model is a more accurate required description.
Thought i never really saw anything like a 750 cant run and a 760/860 can. I saw memory limit of the c700 that a c700 cant run because it get more than 8 megs to run an apps, and the other devices can.
I also saw the situation of a 6000 that cant run stuff where a c7x0 can. Not sure only "processors and resolutions" is a good idea.
I would see something like" Work on: c7x0, c8x0, 5000, 5500 models"
"Sharp Rom, Cacko ROM, tkcrom, OpenZaurus"
or "Tested on xxx,xxx,xxx"
-
That's going to get even more confusing I reckon, especially as the number of ROM releases increases.
Saying that something works on such and such a ROM is all well and good, but does the average punter know which other ROMs it will therefore be compatible with? I doubt it.
That's why I think processor (or even better instruction set - ARM4/ARM5), screen size and dependencies (mainly libc version) are the way to go. It might be possible for the front page to have a list of these parameters for each model, or for some kind of comparison to be performed (choose your model -> it'll tell you whether it'll work, and if not, why not, etc.).
Si
-
I think we have two groups with different aspects and needs.
One is the plain user - he can easily identify:
* which model
* which resolution
* which ROM installed (there is only one and heshe must be aware of)
but typically not:
* library
* processor type
* memory available
The other is the developer of the software - he knows:
* which compiler (2.95.3 or 3.x) and other tools
* which libraries are required
* which model tested on
* which resolution (or independent)
* memory demand
* which ipk version
* which ROM(s) installed and tested
* well, and I doubt a little that heshe really knows the Processor model unless a kernel programmer
Most parts, I think can be solved by a table that allows for translation (e.g. device model to memory size, resolution, processor). And BTW - why should I type that into a database as a developer if it is directly fixed by the device model or the ROM name?
So, more generally, we need to specify the "Platform" the software runs on. Technically it is indeed defined by:
* libraries (APIs) - defined partially by ROM and other installations
* processor architecture and available devices and their features (e.g. screen size)
The first one is mostly defined by e.g. Cacko-1.2.3 and the second one mostly by e.g. SL5500G.
So, my credo is: keep it simple and obvoius for the average user and not necessarily for the developer (he is far ahead in knowledge about the internal design of a Linux PDA).
-- hns
-
Okay, I'll admit defeat ;-)
Will the ROM list be editable?
I think the required libs should be for everyone (as even the newbies will need to get hold of them), but this is not stuff like libc, more like libpcap etc.
If the downloader could put in their ROM type and the site will automatically compare the know details (of the ROM) to the details of the package, that would solve all the problems imo. This requires that the developer is resonably specific about the package (libc, GCC version if it uses C++ libs, screen size limits, etc.), but it's a one-off so not too bad.
Then again it's extra work (but probably worth it to make it easier for the end user).
Si
P.S. I'd be happy to help collating the info and working out what should and shouldn't be able to run on which ROMs, etc.
-
Will the ROM list be editable?
I think the required libs should be for everyone (as even the newbies will need to get hold of them), but this is not stuff like libc, more like libpcap etc.
If the downloader could put in their ROM type and the site will automatically compare the know details (of the ROM) to the details of the package, that would solve all the problems imo. This requires that the developer is resonably specific about the package (libc, GCC version if it uses C++ libs, screen size limits, etc.), but it's a one-off so not too bad.
Then again it's extra work (but probably worth it to make it easier for the end user).
Si
P.S. I'd be happy to help collating the info and working out what should and shouldn't be able to run on which ROMs, etc.
Aye, the ROM list is editable. Nothing is static.
I don't see a point in making anything static as it would just take away from the whole site and in a month make it obsolete.
lm: So what you are proposing, is when someone goes to download a project, they are first promped to enter what type of Rom they are using. Once they enter that, the site itself cross-references what libs that specific Rom is using and what libs the program uses, and then says whether or not they'd need to download additional libraries?
Hmm.. sounds very interesting. Indeed it would be much more work. By doing that, we now require the programmers to enter almost every lib their program uses. I'd imagine though if you are making a program, you know exactly what it's using. As well, we need to get information on any available rom we choose to support. We need to figure out what lib they come with, and what version that lib is.
I have no problems programming this in. It might take me some time, as I need to figure out an efficient way to manage all this. It is possible though; nothing in programming isn't and if we're going to make this thing, there's no point in holding anything back.
lm, perhaps you could get some people to help you gather the required information we'd need for this to work so you don't have to do it all on your own.
One question I have though is would we need the user to also enter their Zaurus model then? Things like screen-size, memory, and processor do make a difference I'd assume.
-
One question I have though is would we need the user to also enter their Zaurus model then? Things like screen-size, memory, and processor do make a difference I'd assume.
Certainly as I understand it some apps written for the 5xxx range do not work with the C models (and presumably the 6000 models) so having the ability to limit the search for programs that will work with the user's model sounds good to me.
Only concern about getting the user to enter their ROM details - what happens when a new ROM comes out - say Cacko 1.22 gets released - will the user suddenly see no apps that will work until someone tests and updates the list?? (This might not be a bad idea, but I guess if so it needs clearly explaining on the site so a new user would know to try searching based on the previous version of the ROM - although we have to be careful with that too!)
-
Only concern about getting the user to enter their ROM details - what happens when a new ROM comes out - say Cacko 1.22 gets released - will the user suddenly see no apps that will work until someone tests and updates the list?? (This might not be a bad idea, but I guess if so it needs clearly explaining on the site so a new user would know to try searching based on the previous version of the ROM - although we have to be careful with that too!)
Well what will happen is we will need to talk to the people who release Cacko and ask them for the list of libraries in this rom and then simply enter it.
The user is still free to download the rom that he or she chooses. They'll simply get a warning telling them that the site is not sure whether or not the rom is supported rather than a list of other libraries they need to download.
-
Well what will happen is we will need to talk to the people who release Cacko and ask them for the list of libraries in this rom and then simply enter it.
Yep, when a ROM is released get the authors to state their version of GCC and libc (which are the two main ROM dependant things I think).
lm: So what you are proposing, is when someone goes to download a project, they are first promped to enter what type of Rom they are using. Once they enter that, the site itself cross-references what libs that specific Rom is using and what libs the program uses, and then says whether or not they'd need to download additional libraries?
I wasn't thinking of cross-referencing all the libs (like the output from ldd), just the specific dependencies which are listed in the control file (though if you could implement it using the output from ldd would be pretty cool :-))
Hmm.. sounds very interesting. Indeed it would be much more work. By doing that, we now require the programmers to enter almost every lib their program uses. I'd imagine though if you are making a program, you know exactly what it's using.
See above, either the control file or ldd are quick and easy, not much for the programmer to do (other than state their version of libc - if you decide on the control file method - and their version of GCC).
As well, we need to get information on any available rom we choose to support. We need to figure out what lib they come with, and what version that lib is.
Again, it depends how far you want to go with it: One option is to just list the libc version (and possibly other must-haves such as libncurses); the other, is of course, to go the whole hog and have a db containing a list of every standard lib in the image and their versions (not sure how much work this would be, but probably not insurmountable, assuming new ROMs aren't released daily ;-)).
One question I have though is would we need the user to also enter their Zaurus model then? Things like screen-size, memory, and processor do make a difference I'd assume.
Yes. The ROM defines some of the constraints, but the model defines the others (instruction set and screen size), so both would be needed (and people would know both too).
Si
P.S. Perhaps I'm going too far here? I'm quite happy to read a list of the ROMs which people have tried, and to make my own mind up, but for those with less experience I think this system would be hassle-free (though possibly not for the maintainers). Feel free to shoot me down on any of this (especially if you think I'm going to far - I am a bit of a perfectionist)
-
It would not be possible to do a script
that read the ipk, extract the file
do a ldd on the executable, and see the depencies
and automaticaly list the librairies in the project ?
I know i ask for a lots =P
Nice job btw, i saw the screenshots. Its going to be great to see the front page result with the new items etc kinda like freshmeat
-
...and I'm sure someone who's better at shell scripts than I am could come up with something which does a 'ls /lib/*.so.?.?' and 'ls /usr/lib*.so.?.?' and collates the info for the standard-libs-in-such-and-such-a-ROM database for the above to be compared against.
(dz - beginning to regret taking this on? )
Si
-
Heh, no i don't regret it. This is a fun project
As far as making a script that reads the file, the only problem I see with that is server load.
I'm not exactly positive of the structure of IPK's, but even if I were to extract it, how would I know which file to check? Is there some kind of way to find out?
-
I'm not exactly positive of the structure of IPK's, but even if I were to extract it, how would I know which file to check? Is there some kind of way to find out?
If you're thinking of the control file, it's always the only file in the control.tar.gz file which is inside the ipk.
If you're wondering which file to look at and run ldd on, that's a bit more compilcated. A couple of ideas: extract the data.tar.gz component of the ipk, then look at all files in any bin directory (because they could be /bin, /opt/QtPalmtop/bin, /usr/bin etc.), might also be worth looking inside lib directories too if there are no bin directories; check the file permissions.
As far as making a script that reads the file, the only problem I see with that is server load.
Assuming you create a db rather than reading the file each time the data is requested it shouldn't be to bad (I reckon); unless the ipk submission rate goes up by a very large margin ;-)
Si
-
Addendum:
libc etc. are of the format lib*-?.?.?.so, so this would have to be added and searched for too.
ldd only returns the major versions of the libs which the binary is linked against. This is not supposed to be a problem (e.g. minor version changes aren't supposed to break things - at least that's what I understood), but can be sometimes. Libc version cannot be ascertained from ldd (unless the random numbers after the lib name actually mean something to somebody?).
Si
-
Yep, when a ROM is released get the authors to state their version of GCC and libc (which are the two main ROM dependant things I think).
This mainly recommends three (or four) different categories of software:
1. applications (.ipk) - ask for Model, ROM
2. ROMs - ask for Model, CPU type, gcc, libc, Memory, anything else
3. libraries
4. others
Having that it would even allow the maintainer of a ROM to add all the information by themselves.
Just another idea.
-- hns
-
As far as making a script that reads the file, the only problem I see with that is server load.
The server would only do it once. The way i saw it, its when someone upload an .ipk, the server offer him a list of libraries that his binary contain kinda like
Extract the .ipk
run ldd on the "binary" it to see the libs (might be tricky to fidn the binary
and return the result on the web page. So the author can edit them.
Not sure if its really required anyway, not like the author dont include the libs, or tell the dependancy needed "The project require libsdl and libmad".
etc...
-
Ok, how about this. How about we formulate a list of libs, and when you are submitting a project, you are shown this list and are asked to select all the libs involved in your project.
That way, I dont have to mess with extracting IPK's and all that. I dont know how to tell which file i the executable or not, so I figure this way is the easiest.
I need a list of most libraries though. As well, if the user uses other libraries, they can request it added to the list.
That work?
-
Yes, thats actually sound a good way to do it
Maybe have version number and a link to download the library ipk if possible ?:
I know its a lots of more work. Not sure if its really gonna be usefull but something like that that i would see.
Requirement
the_frog_version_1.0.ipk
by Mr Froggy.
Work on: Cacko ROM, tckrom, Sharp ROM, OpenZaurus
Supported resolution(s): 640x480
Supported model(s): SL-C700, SL-C750, SL-C780, SL-6000
Processor(s): ARM5 (pxa250, pxa255)
----
This project include ALL libraries already needed to run this program.
----
This project require the following libraries (not included) to be installed on your ROM.
libsdl ÂÂ>= 1.2.5
libpcap >= 1.0.3
libfreetype >= 2.4.5
The project include the following libraries:
libmad 0.41
libssl x.x.xx
----
Cacko ROM 1.22b include the following libraries:
libfreetype 2.6.5
libpcap 1.3.4
Sharp ROM doesn't have any required libraries
Tck ROM include the following libraries:
libfreetype 2.6.6
OpenZaurus ROM include all libraires needed.
-----
I dont know if its possible to have a list of libraries by rom but thats would be something similar.
-
I accept that you don't want the extra work, but for completeness...:
I dont know how to tell which file i the executable or not, so I figure this way is the easiest.
Look in all bin directories at all files which 'file' considers to be executables (or parse the ls -rtl data and see whether they are executable). May as well look in lib directories too if there are no bin directories ;-)
Nevertheless, some dependencies will be missed this way no matter what happens:- for example python scripts won't work without the runtime, but this wouldn't be picked up using the above to analyse the scripts (unless you want to get really complicated and also analyse the hash-bang in executable text files..... no sounds like a lot of work to me ;-))
I need a list of most libraries though. As well, if the user uses other libraries, they can request it added to the list.
Perhaps they should just add the info manually then rather than having a list (which will quickly grow rather large). Assuming the comments can be made for the entries, then the correct libs will soon be listed (by the author reading the comments and changing the main page, etc.).
Si
-
The project include the following libraries:
libmad 0.41
libssl x.x.xx
This, in itself, is somewhat problematic. I'd prefer it if people would just package up their program and list the dependencies rather than adding extra libs (which may overwrite the ones which I already have).
I dont know if its possible to have a list of libraries by rom but thats would be something similar.
This is relatively trivial, just run something like this on a clean install of each ROM (off the top of my head):
#!/bin/bash
# parse all libs on a Z
if [$1 = ]; then
echo "no files given"
exit 0
fi
for i in $( ls /lib/lib*.so.?.? ); do
echo item: $i
done
for i in $( ls /lib/lib*-?.?.?.so ); do
echo item: $i
done
for i in $( ls /usr/lib/lib*.so.?.? ); do
echo item: $i
done
for i in $( ls /usr/lib/lib*-?.?.?.so ); do
echo item: $i
done
# need to stick this info into a db
# check there are no duplicates first...?
-
It's not really a matter of extra work, but more-so a matter of I want to make a way for the zsi2 to be positive of what the program needs. Like you said, what happens if the zsi2 misses the binary and therefore lists no required libs. By having the programmer do it as he's submitting his project and we know exactly what it needs because he tells us.
I do agree with not packaging libs, and as far as I was aware, I didnt know programs actually did package libs alongside.
I dont know if its possible to have a list of libraries by rom but thats would be something similar.
Why can't we simply asking the rom makers what libs they included? Roms are packed tight, and when you're making one, I'd imagine you know practically every piece of software in it.
-
Just wanted to let you all know that I haven't forgotten about this, and I have been working on it. I just finished up the 'add release' portion. I'll explain how it works in detail once I get more done.
I'm hoping to be able to release it for testing soon. I need to add some more error checking to the 'add release' portion, and then i have a small list of things left to do before live release. Live release though will not mean fully functional release. It will simply be to test all the current features and work out the current bugs.
Fully functional release though will be soon
-
Cool, keep up the good work.
Si
-
Well done dz, I'm looking forward to seeing the progress. This is an exciting prospect for Z users everywhere