modred team mailing list archive

[Frederic Koehler <fkfire@xxxxxxxxx>]

The idea for client-side computation implies that we have highly-trusted
clients... (we know they won't provide invalid answers) Otherwise,
client-side computation requires verifying answers and so is only useful for
a few NP-ish problems. In addition (assuming trusted clients).

Also, it means that, since computations can contain sensitive data, the
abillity to spread the computation is limited - unless we know the
computation is not user-sensitive, it can only try to use the user's
client(s). This way we also know that the client has no interest in
sabatoging answers to mess with other users (except to exploit server-side
weaknesses, which is inevitable).

On Sat, Dec 26, 2009 at 10:28 PM, Scott Lawrence <bytbox@xxxxxxxxx> wrote:

> Here is what, as I envision it, will make modred unique (and hard):
>
>  * Support for clients who can come and leave, lending CPU time and
> using CPU time as they choose.  There are some clusters that support
> this, but not very many.
>  * Support for computers participating across the internet.  This goes
> along with the previous part, but remember we need security to make
> this worth anything. This also means that user data could potentially
> be passed to untrusted computers - we need a way to prevent this.
>  * The ability for clients to run on any OS, using perl, python, java,
> or (on unix systems) C and C++ (servers and the hub will need to run
> on linux or at least another unix, or a dedicated OS which we may
> decide to write)
>  * Modred has great ease of use because it acts as a single unified
> computer - a special client program exists that allows one to log in,
> access and edit files, etc...  This is very close to unique - google
> has it, though
>
> Because of that last point, many OS design issues should come up when
> we code modred. (I think Freddy pointed this out?) Thus, we have a
> chance to fix flaws in standard unix, incorporating plan 9-type stuff
> (google it and read about it - Plan 9 from Bell Labs, the way the
> future of unix was) while also creating an actually usable user
> interface. (No offense, but to a newbie non-super-technical user,
> linux is a bit harsh...)
>
> Some implementation questions and ideas:
>
>  - how will updates be handled?  Remember we've got 200 computers
> potentially, some of which might be clients that want to participate
> in multiple clusters.
>
>  - maybe we should have programs not include front ends.  Instead, the
> modred software creates a front-end from the program's self
> description.  This would enforce a consistent user interface if we
> could implement it well
>
>  - how can we keep users from being able to snoop on each others' data?
>
> That's just a sample to get people thinking.
>
>
> On 12/26/09, David Tolnay <dtolnay@xxxxxxxxx> wrote:
> > Before diving in to specifics about the implementation I think we need
> > to decide how we want modred to be different from (read: better than)
> > existing bootable cluster environments. Here is a short list to check
> > out:
> >
> > Bootable Cluster CD (http://bccd.net/) - folks presented this at SC09
> > in portland, it was pretty neat stuff. Packed with education /
> > debugging / visualization features
> >
> > Oscar (http://svn.oscar.openclustergroup.org/trac/oscar) - very
> > trivially simple way to transform an existing unix lab into a cluster
> > resource
> >
> > Lnx-bbc (http://www.lnx-bbc.com/) - includes cowsay!
> >
> > Perceus/warewulf (http://www.perceus.org/portal/) - a lot of other
> > sites made reference to this, haven't read too much about it
> >
> > What specifically do you want to improve over any of these?
> >
> >
> > On 12/25/09, Frederic Koehler <fkfire@xxxxxxxxx> wrote:
> >>
> >>
> >> So, as far as I understand this project, the idea is to build
> >> both a client library and a program using the library to do clustering
> >> stuff, along with matching server/hub foo (the library might be the same
> >> or
> >> whatever, not important).
> >>
> >> So from this understanding, it seems that the system should provide some
> >> basic pseudo-operating system stuff and programs can build on that, just
> >> like they would normally build on their local libc/kernel and stuff.
> >>
> >> So (I sure like the word "so" today...) if we want the type of general
> >> os-like stuff it seems their needs to be support for:
> >>    * A simpe message passing model - abstract away all the TCP-foo,
>  maybe
> >> use existing foo here (obviously needs fleshing out)
> >>    * Permanent storage IO (clone the unix write(), read(), open() and
> >> sync()
> >> model,  or maybe just use one of the existing database-ish nosql things
> >> out
> >> there)
> >>            - Unix-ish model - you create your data hunk, say you want
> all
> >> this stuff in it, then after sync() we know it's actually somewhere
> >> written
> >> on a hard-drive, and other things can read it too
> >>            - Unless this isn't in fact needed (but I assume it is)
> >>            - Also need to figure out if it's filesystem-ish foo
> >> (hierarchial) we want or more relational database-ish stuff
> >>
> >>    * A task delegation model - some type of map/reduce-ish stuff
> >>           - Servers have a few built-in computations, and client
> utilizes
> >> them?
> >>           - Or more complex, servers run sandboxed computational code?
> >>    * A security system?
> >>         - Needs fleshing out
> >>         - Presumably what the "hub" manages - it's the trusted thing
> >>         - Obviously, not everybody is allowed to use the cluster for
> >> computation, not everybody can find out what everybody else is doing,
> etc.
> >>       - But also, is their a limit on storage, are some things
> prioritized
> >> over others, ?
> >>
> >> Theroretically, server's are written to provide the io backend and to
> >> allow
> >> for task delegation, clients use the api, although hub has it's work cut
> >> out
> >> delegating all the file io and figuring out what the state of that is.
> >>
> >> On top of some mixture of this, one could build a simple unix-ish
> >> pseudo-cli, theroretically, as well as real software.
> >>
> >> Anyway, before actually doing anything, people should read about PVM
> >> (Parallel Virtual Machine) and the like (maybe also Hadoop and other
> >> foo-ish
> >> stuff) so Modred isn't just a bad clone of it
> >>
> >> Anyway, (yes, twice in a row!), I figured _someone_ had to respond to
> >> Scott,
> >> otherwise he'd feel all lonely and sad :P Now he can have a warm fuzzy
> >> feeling of deep confusion and uncertainty instead :P
> >>
> >>
> >>
> >> On Fri, Dec 25, 2009 at 11:06 PM, Scott Lawrence <bytbox@xxxxxxxxx>
> wrote:
> >> > ---------- Forwarded message ----------
> >> > From: Scott Lawrence <bytbox@xxxxxxxxx>
> >> > Date: Fri, 25 Dec 2009 19:20:13 -0500
> >> > Subject: Design Overview
> >> > To: modred <modred@xxxxxxxxxxxxxxxxxxx>
> >> >
> >> > I'm going to assume that everyone understands the basic concepts for
> >> > modred: a set of networked computers (by 'networked' I mean, they're
> >> > all on the internet), divided for the sake of discussion into three
> >> > classes: the 'hub' (the dude in charge, who compupters who want to
> >> > join connect to), the 'servers' (dedicated computers that can be
> >> > pretty much relied on not to go down, although redundancy is always
> >> > nice), and the 'clients' (computers that send in requests and can be
> >> > used for spare CPU cycles.
> >> >
> >> > Ok, so much for assumptions... :-)
> >> >
> >> > Things *I* think any design should emphasize:
> >> >  * security.
> >> >  * relative ease of use, while retaining significant power.
> >> > Challenging.  In particular, it should be possible to set up a modred
> >> > network in under an hour, provided the computers are already set up.
> >> >  * along with the previous bullet point, having an interface that lets
> >> > one use the entire network like a single computer.  This is sort of
> >> > like the way google docs works, except the cloud is private
> >> >  * therefore, it should be a multi-user system with well-designed
> >> > privileges etc...
> >> >
> >> > I'm not going to discuss my implementation ideas, let's hear others
> >> > first.
> >> >
> >> > --
> >> > Scott Lawrence
> >> >
> >> > Webmaster
> >> > The Blair Robot Project
> >> > Montgomery Blair High School
> >> >
> >> >
> >> >
> >> > --
> >> > Scott Lawrence
> >> >
> >> > Webmaster
> >> > The Blair Robot Project
> >> > Montgomery Blair High School
> >> >
> >>
> >>
> >>
> >> _______________________________________________
> >>  Mailing list: https://launchpad.net/~modred
> >>  Post to     : modred@xxxxxxxxxxxxxxxxxxx
> >>  Unsubscribe : https://launchpad.net/~modred
> >>  More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~modred
> > Post to     : modred@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~modred
> > More help   : https://help.launchpad.net/ListHelp
> >
>
>
> --
> Scott Lawrence
>
> Webmaster
> The Blair Robot Project
> Montgomery Blair High School
>
> _______________________________________________
> Mailing list: https://launchpad.net/~modred
> Post to     : modred@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~modred
> More help   : https://help.launchpad.net/ListHelp
>