modred team mailing list archive

[Scott Lawrence <bytbox@xxxxxxxxx>]

Here is what, as I envision it, will make modred unique (and hard):

 * Support for clients who can come and leave, lending CPU time and
using CPU time as they choose.  There are some clusters that support
this, but not very many.
 * Support for computers participating across the internet.  This goes
along with the previous part, but remember we need security to make
this worth anything. This also means that user data could potentially
be passed to untrusted computers - we need a way to prevent this.
 * The ability for clients to run on any OS, using perl, python, java,
or (on unix systems) C and C++ (servers and the hub will need to run
on linux or at least another unix, or a dedicated OS which we may
decide to write)
 * Modred has great ease of use because it acts as a single unified
computer - a special client program exists that allows one to log in,
access and edit files, etc...  This is very close to unique - google
has it, though

Because of that last point, many OS design issues should come up when
we code modred. (I think Freddy pointed this out?) Thus, we have a
chance to fix flaws in standard unix, incorporating plan 9-type stuff
(google it and read about it - Plan 9 from Bell Labs, the way the
future of unix was) while also creating an actually usable user
interface. (No offense, but to a newbie non-super-technical user,
linux is a bit harsh...)

Some implementation questions and ideas:

 - how will updates be handled?  Remember we've got 200 computers
potentially, some of which might be clients that want to participate
in multiple clusters.

 - maybe we should have programs not include front ends.  Instead, the
modred software creates a front-end from the program's self
description.  This would enforce a consistent user interface if we
could implement it well

 - how can we keep users from being able to snoop on each others' data?

That's just a sample to get people thinking.


On 12/26/09, David Tolnay <dtolnay@xxxxxxxxx> wrote:
> Before diving in to specifics about the implementation I think we need
> to decide how we want modred to be different from (read: better than)
> existing bootable cluster environments. Here is a short list to check
> out:
>
> Bootable Cluster CD (http://bccd.net/) - folks presented this at SC09
> in portland, it was pretty neat stuff. Packed with education /
> debugging / visualization features
>
> Oscar (http://svn.oscar.openclustergroup.org/trac/oscar) - very
> trivially simple way to transform an existing unix lab into a cluster
> resource
>
> Lnx-bbc (http://www.lnx-bbc.com/) - includes cowsay!
>
> Perceus/warewulf (http://www.perceus.org/portal/) - a lot of other
> sites made reference to this, haven't read too much about it
>
> What specifically do you want to improve over any of these?
>
>
> On 12/25/09, Frederic Koehler <fkfire@xxxxxxxxx> wrote:
>>
>>
>> So, as far as I understand this project, the idea is to build
>> both a client library and a program using the library to do clustering
>> stuff, along with matching server/hub foo (the library might be the same
>> or
>> whatever, not important).
>>
>> So from this understanding, it seems that the system should provide some
>> basic pseudo-operating system stuff and programs can build on that, just
>> like they would normally build on their local libc/kernel and stuff.
>>
>> So (I sure like the word "so" today...) if we want the type of general
>> os-like stuff it seems their needs to be support for:
>>    * A simpe message passing model - abstract away all the TCP-foo,  maybe
>> use existing foo here (obviously needs fleshing out)
>>    * Permanent storage IO (clone the unix write(), read(), open() and
>> sync()
>> model,  or maybe just use one of the existing database-ish nosql things
>> out
>> there)
>>            - Unix-ish model - you create your data hunk, say you want all
>> this stuff in it, then after sync() we know it's actually somewhere
>> written
>> on a hard-drive, and other things can read it too
>>            - Unless this isn't in fact needed (but I assume it is)
>>            - Also need to figure out if it's filesystem-ish foo
>> (hierarchial) we want or more relational database-ish stuff
>>
>>    * A task delegation model - some type of map/reduce-ish stuff
>>           - Servers have a few built-in computations, and client utilizes
>> them?
>>           - Or more complex, servers run sandboxed computational code?
>>    * A security system?
>>         - Needs fleshing out
>>         - Presumably what the "hub" manages - it's the trusted thing
>>         - Obviously, not everybody is allowed to use the cluster for
>> computation, not everybody can find out what everybody else is doing, etc.
>>       - But also, is their a limit on storage, are some things prioritized
>> over others, ?
>>
>> Theroretically, server's are written to provide the io backend and to
>> allow
>> for task delegation, clients use the api, although hub has it's work cut
>> out
>> delegating all the file io and figuring out what the state of that is.
>>
>> On top of some mixture of this, one could build a simple unix-ish
>> pseudo-cli, theroretically, as well as real software.
>>
>> Anyway, before actually doing anything, people should read about PVM
>> (Parallel Virtual Machine) and the like (maybe also Hadoop and other
>> foo-ish
>> stuff) so Modred isn't just a bad clone of it
>>
>> Anyway, (yes, twice in a row!), I figured _someone_ had to respond to
>> Scott,
>> otherwise he'd feel all lonely and sad :P Now he can have a warm fuzzy
>> feeling of deep confusion and uncertainty instead :P
>>
>>
>>
>> On Fri, Dec 25, 2009 at 11:06 PM, Scott Lawrence <bytbox@xxxxxxxxx> wrote:
>> > ---------- Forwarded message ----------
>> > From: Scott Lawrence <bytbox@xxxxxxxxx>
>> > Date: Fri, 25 Dec 2009 19:20:13 -0500
>> > Subject: Design Overview
>> > To: modred <modred@xxxxxxxxxxxxxxxxxxx>
>> >
>> > I'm going to assume that everyone understands the basic concepts for
>> > modred: a set of networked computers (by 'networked' I mean, they're
>> > all on the internet), divided for the sake of discussion into three
>> > classes: the 'hub' (the dude in charge, who compupters who want to
>> > join connect to), the 'servers' (dedicated computers that can be
>> > pretty much relied on not to go down, although redundancy is always
>> > nice), and the 'clients' (computers that send in requests and can be
>> > used for spare CPU cycles.
>> >
>> > Ok, so much for assumptions... :-)
>> >
>> > Things *I* think any design should emphasize:
>> >  * security.
>> >  * relative ease of use, while retaining significant power.
>> > Challenging.  In particular, it should be possible to set up a modred
>> > network in under an hour, provided the computers are already set up.
>> >  * along with the previous bullet point, having an interface that lets
>> > one use the entire network like a single computer.  This is sort of
>> > like the way google docs works, except the cloud is private
>> >  * therefore, it should be a multi-user system with well-designed
>> > privileges etc...
>> >
>> > I'm not going to discuss my implementation ideas, let's hear others
>> > first.
>> >
>> > --
>> > Scott Lawrence
>> >
>> > Webmaster
>> > The Blair Robot Project
>> > Montgomery Blair High School
>> >
>> >
>> >
>> > --
>> > Scott Lawrence
>> >
>> > Webmaster
>> > The Blair Robot Project
>> > Montgomery Blair High School
>> >
>>
>>
>>
>> _______________________________________________
>>  Mailing list: https://launchpad.net/~modred
>>  Post to     : modred@xxxxxxxxxxxxxxxxxxx
>>  Unsubscribe : https://launchpad.net/~modred
>>  More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~modred
> Post to     : modred@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~modred
> More help   : https://help.launchpad.net/ListHelp
>


-- 
Scott Lawrence

Webmaster
The Blair Robot Project
Montgomery Blair High School