modred team mailing list archive

[Frederic Koehler <fkfire@xxxxxxxxx>]

Well , I don't consider CPU time credits to be that important - they're only
really needed to avoid allocating too much time in the case of a very short
request vs. a long one, neither of which is really a good situation anyway.

However, I guess a less horrible mechanism for small clusters would be to
just time (like userspace time + kernel time, not real time) a few runs on
the servers.

On Mon, Dec 28, 2009 at 10:09 PM, Michael Cohen <gnurdux@xxxxxxxxx> wrote:

> > * CPU time credits can be very roughly estimated by averaging response
> time.
> > It's not all _that_ important anyway if nothing is a behemoth task.
>
> In my opinion, that's worse than not doing it at all.  That means that if a
> client has more stuff running, so that the program doesn't get 90% of the
> CPU, then he gets more credits.  That's dumb.
>
> Michael Cohen
>
> Frederic Koehler wrote:
>
>> Hah, my email died, wow...Wonder what happened to it....
>> Anyway, this is something like what I wrote before:
>>
>> * CPU time credits can be very roughly estimated by averaging response
>> time.
>> It's not all _that_ important anyway if nothing is a behemoth task.
>>
>> * The hub can immediately, upon establishing connection, redirect client
>> to
>> a server. The server will still have to communicate with hub somewhat, but
>> it can only send stuff necessarily pertaining to the hub.
>>
>> * Jobs should probably be many small tasks to avoid the risk of losing a
>> giant computation (since saving computation state is not
>> easy/generalizable). Beyond that, sending keep-alive packets is enough to
>> know when a client dies.
>>
>>
>> On Mon, Dec 28, 2009 at 6:16 PM, Scott Lawrence <bytbox@xxxxxxxxx> wrote:
>>
>>  What?
>>>
>>> On 12/28/09, Frederic Koehler <fkfire@xxxxxxxxx> wrote:
>>>
>>>> On Mon, Dec 28, 2009 at 12:45 AM, Scott Lawrence <bytbox@xxxxxxxxx>
>>>>
>>> wrote:
>>>
>>>> "What happens when a client disconnects with unfinished work? Is the
>>>>> work immeditately reassigned, or does the server wait for a specified
>>>>> period, etc. This could come up quite a lot because some clients will
>>>>> just disconnect as soon as work they submitted is completed."
>>>>>
>>>>> Ouch.  Good question.  Here's one solution: small tasks (expected time
>>>>> <2 seconds) are always assigned to two or more clients/servers.  If
>>>>> both disconnect, reassign, if one disconnects, use the other guy's
>>>>> answer. Large tasks, if a computer stops regularly checking in every 5
>>>>> or so seconds, give that computer's results to date to another
>>>>> computer.  So yeah, I think a client should have to make regular
>>>>> reports to a server.
>>>>>
>>>>> Here's another problem: how do we tell how many CPU time credits to
>>>>> grant a client?  We can't always tell how long a problem should take
>>>>> beforehand.
>>>>>
>>>>> Here's another problem: which computer should handle the clients?  As
>>>>> I've been thinking about this, there are three types of computers, the
>>>>> single hub, the various dedicated servers (capable of storing
>>>>> permanent data), and the clients.  (The hub is necessary - without it,
>>>>> the performance of the cluster drastically decreases.) So clients
>>>>> connect to the hub, and then the hub directs all computers.  But the
>>>>> hub will get overloaded if 100 computers are checking in every 10
>>>>> seconds to give it more data (and then the hub has to pass this on to
>>>>> other servers for storage, etc...).  So at some point, the hub needs
>>>>> to tell the client to talk to the server.  When?
>>>>>
>>>>> Who wants to create that prototype?
>>>>>
>>>>> On 12/28/09, Scott Lawrence <bytbox@xxxxxxxxx> wrote:
>>>>>
>>>>>> This is where we start building prototypes. However, just to keep the
>>>>>> theoretical side going: I disagree about the privacy issue.  Most
>>>>>> operations that would benefit from the CPU time of a cluster (notice
>>>>>> I'm not talking about the data storage and reliability benefits, which
>>>>>> aren't affected by the presence of clients) are not very private.
>>>>>> Rendering nice screensavers ("Electric Sheep", I think that one's
>>>>>> called), and hefty data sifting aren't private - who cares about the
>>>>>> screensavers, and the data is generally public anyway (of course if it
>>>>>> wasn't, it would be marked so).
>>>>>>
>>>>>> Ray tracing and simulation could be more of an issue.  Hypothetical
>>>>>> situation: Alice is simulating how wind will affect her proprietary
>>>>>> airplane design.  Naturally, she can't hand off the whole design, or
>>>>>> even parts of the design, to random client computers.  This is where
>>>>>> the windows programmer says, "so the client computers can't help
>>>>>> Alice."  But that's not true - as a bad example, what if the Modred
>>>>>> hub gave to a client computer 80 types of landing gear, and told the
>>>>>> computer, not to simulate something, but to solve a general formula
>>>>>> that could later be used in the computation in a trivial and quick
>>>>>> way?  If that client is evil, it will learn that Alice's airplane has
>>>>>> some sort of landing gear.
>>>>>>
>>>>>> Somebody needs to create a prototype of a server that can create
>>>>>> arbitrary problems in some format, so we can all try to trick it.  I
>>>>>> suggest lisp as the language, but it's up to the implementer.
>>>>>>
>>>>>>
>>>>>> On 12/28/09, Scott Lawrence <bytbox@xxxxxxxxx> wrote:
>>>>>>
>>>>>>> ---------- Forwarded message ----------
>>>>>>> From: Frederic Koehler <fkfire@xxxxxxxxx>
>>>>>>> Date: Sun, 27 Dec 2009 23:21:28 -0500
>>>>>>> Subject: Re: [Modred] Concept stuff
>>>>>>> To: Scott Lawrence <bytbox@xxxxxxxxx>
>>>>>>>
>>>>>>>  * This is sort-of a solution (while obviously less-than-optimal
>>>>>>> security,
>>>>>>> some grid-computing stuff does this, like BOINC), however, it turns
>>>>>>>
>>>>>> out
>>>
>>>> that
>>>>>>> this may require custom validation methods - for example, it's normal
>>>>>>>
>>>>>> for
>>>>>
>>>>>> floating point values to be different on different computers, and the
>>>>>>> same
>>>>>>> could apply for other computations.
>>>>>>>  * A malicious client would only need to misbehave on certain
>>>>>>>
>>>>>> problems
>>>
>>>> that
>>>>>>> a malicious user could designate (or recognize obvious fake
>>>>>>>
>>>>>> programs),
>>>
>>>> allowing the fake program test to work.
>>>>>>>    - A better idea would be to randomly reduplicate some
>>>>>>>
>>>>>> computations
>>>
>>>> many
>>>>>>> times - the malicious client wouldn't notice anything, but could
>>>>>>>
>>>>>> easily
>>>
>>>> be
>>>>>>> singled out
>>>>>>>
>>>>>>>  * Thirdly is mostly the same thing I wrote before - only
>>>>>>>
>>>>>> computations
>>>
>>>> that
>>>>>>> are said to be totally unimportant privacy wise could benefit from
>>>>>>> client-side computing.
>>>>>>>
>>>>>>> So I really think that client-side computation is only a good idea
>>>>>>>
>>>>>> for
>>>
>>>> a
>>>>>>> small subset of problems (like the type that there already exist
>>>>>>> massive
>>>>>>> grid computing solutions for, like SETI@HOME)
>>>>>>>
>>>>>>> On Sun, Dec 27, 2009 at 10:57 PM, Scott Lawrence <bytbox@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  I want clients to be used for computation, and I want maximum
>>>>>>>> privacy+security given that restriction.  Some ideas:
>>>>>>>>
>>>>>>>> With a large network, two computers can perform the same
>>>>>>>>
>>>>>>> computation.
>>>
>>>> Furthermore, a smart modred hub can give fake problems to clients,
>>>>>>>> just to make sure that they're operating correctly.  A client that
>>>>>>>> isn't operating correctly gets cut. (No second chances! A program
>>>>>>>> could exploit that!)
>>>>>>>>
>>>>>>>> If a user specifies a certain bit of data (SSN, for instance) as
>>>>>>>> highly sensitive, modred should know not to hand off that
>>>>>>>>
>>>>>>> computation
>>>
>>>> to a client. (If it does by accident, it certainly should never hand
>>>>>>>> off the data.) privacy++
>>>>>>>>
>>>>>>>> In all cases, computations should be anonymous. privacy++
>>>>>>>>
>>>>>>>> Other ideas?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12/27/09, Frederic Koehler <fkfire@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> The idea for client-side computation implies that we have
>>>>>>>>> highly-trusted
>>>>>>>>> clients... (we know they won't provide invalid answers) Otherwise,
>>>>>>>>> client-side computation requires verifying answers and so is only
>>>>>>>>> useful
>>>>>>>>>
>>>>>>>> for
>>>>>>>>
>>>>>>>>> a few NP-ish problems. In addition (assuming trusted clients).
>>>>>>>>>
>>>>>>>>> Also, it means that, since computations can contain sensitive
>>>>>>>>>
>>>>>>>> data,
>>>
>>>> the
>>>>>>>>> abillity to spread the computation is limited - unless we know the
>>>>>>>>> computation is not user-sensitive, it can only try to use the
>>>>>>>>>
>>>>>>>> user's
>>>
>>>> client(s). This way we also know that the client has no interest
>>>>>>>>>
>>>>>>>> in
>>>
>>>> sabatoging answers to mess with other users (except to exploit
>>>>>>>>>
>>>>>>>> server-side
>>>>>>>>
>>>>>>>>> weaknesses, which is inevitable).
>>>>>>>>>
>>>>>>>>> On Sat, Dec 26, 2009 at 10:28 PM, Scott Lawrence <
>>>>>>>>>
>>>>>>>> bytbox@xxxxxxxxx>
>>>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Here is what, as I envision it, will make modred unique (and
>>>>>>>>>>
>>>>>>>>> hard):
>>>
>>>>  * Support for clients who can come and leave, lending CPU time
>>>>>>>>>>
>>>>>>>>> and
>>>
>>>> using CPU time as they choose.  There are some clusters that
>>>>>>>>>> support
>>>>>>>>>> this, but not very many.
>>>>>>>>>>  * Support for computers participating across the internet.  This
>>>>>>>>>> goes
>>>>>>>>>> along with the previous part, but remember we need security to
>>>>>>>>>>
>>>>>>>>> make
>>>
>>>> this worth anything. This also means that user data could
>>>>>>>>>>
>>>>>>>>> potentially
>>>>>
>>>>>> be passed to untrusted computers - we need a way to prevent this.
>>>>>>>>>>  * The ability for clients to run on any OS, using perl, python,
>>>>>>>>>> java,
>>>>>>>>>> or (on unix systems) C and C++ (servers and the hub will need to
>>>>>>>>>> run
>>>>>>>>>> on linux or at least another unix, or a dedicated OS which we may
>>>>>>>>>> decide to write)
>>>>>>>>>>  * Modred has great ease of use because it acts as a single
>>>>>>>>>>
>>>>>>>>> unified
>>>
>>>> computer - a special client program exists that allows one to log
>>>>>>>>>>
>>>>>>>>> in,
>>>>>
>>>>>> access and edit files, etc...  This is very close to unique -
>>>>>>>>>> google
>>>>>>>>>> has it, though
>>>>>>>>>>
>>>>>>>>>> Because of that last point, many OS design issues should come up
>>>>>>>>>>
>>>>>>>>> when
>>>>>
>>>>>> we code modred. (I think Freddy pointed this out?) Thus, we have
>>>>>>>>>>
>>>>>>>>> a
>>>
>>>> chance to fix flaws in standard unix, incorporating plan 9-type
>>>>>>>>>>
>>>>>>>>> stuff
>>>>>
>>>>>> (google it and read about it - Plan 9 from Bell Labs, the way the
>>>>>>>>>> future of unix was) while also creating an actually usable user
>>>>>>>>>> interface. (No offense, but to a newbie non-super-technical user,
>>>>>>>>>> linux is a bit harsh...)
>>>>>>>>>>
>>>>>>>>>> Some implementation questions and ideas:
>>>>>>>>>>
>>>>>>>>>>  - how will updates be handled?  Remember we've got 200 computers
>>>>>>>>>> potentially, some of which might be clients that want to
>>>>>>>>>> participate
>>>>>>>>>> in multiple clusters.
>>>>>>>>>>
>>>>>>>>>>  - maybe we should have programs not include front ends.
>>>>>>>>>>
>>>>>>>>>  Instead,
>>>
>>>> the
>>>>>>>>>> modred software creates a front-end from the program's self
>>>>>>>>>> description.  This would enforce a consistent user interface if
>>>>>>>>>>
>>>>>>>>> we
>>>
>>>> could implement it well
>>>>>>>>>>
>>>>>>>>>>  - how can we keep users from being able to snoop on each others'
>>>>>>>>>> data?
>>>>>>>>>>
>>>>>>>>>> That's just a sample to get people thinking.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 12/26/09, David Tolnay <dtolnay@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> Before diving in to specifics about the implementation I think
>>>>>>>>>>>
>>>>>>>>>> we
>>>
>>>> need
>>>>>>>>>>> to decide how we want modred to be different from (read: better
>>>>>>>>>>> than)
>>>>>>>>>>> existing bootable cluster environments. Here is a short list to
>>>>>>>>>>> check
>>>>>>>>>>> out:
>>>>>>>>>>>
>>>>>>>>>>> Bootable Cluster CD (http://bccd.net/) - folks presented this
>>>>>>>>>>>
>>>>>>>>>> at
>>>
>>>> SC09
>>>>>>>>>>> in portland, it was pretty neat stuff. Packed with education /
>>>>>>>>>>> debugging / visualization features
>>>>>>>>>>>
>>>>>>>>>>> Oscar (http://svn.oscar.openclustergroup.org/trac/oscar) -
>>>>>>>>>>>
>>>>>>>>>> very
>>>
>>>> trivially simple way to transform an existing unix lab into a
>>>>>>>>>>> cluster
>>>>>>>>>>> resource
>>>>>>>>>>>
>>>>>>>>>>> Lnx-bbc (http://www.lnx-bbc.com/) - includes cowsay!
>>>>>>>>>>>
>>>>>>>>>>> Perceus/warewulf (http://www.perceus.org/portal/) - a lot of
>>>>>>>>>>>
>>>>>>>>>> other
>>>>>
>>>>>> sites made reference to this, haven't read too much about it
>>>>>>>>>>>
>>>>>>>>>>> What specifically do you want to improve over any of these?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 12/25/09, Frederic Koehler <fkfire@xxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So, as far as I understand this project, the idea is to build
>>>>>>>>>>>> both a client library and a program using the library to do
>>>>>>>>>>>>
>>>>>>>>>>> clustering
>>>>>>>>
>>>>>>>>> stuff, along with matching server/hub foo (the library might
>>>>>>>>>>>>
>>>>>>>>>>> be
>>>
>>>> the
>>>>>>>>>>>> same
>>>>>>>>>>>> or
>>>>>>>>>>>> whatever, not important).
>>>>>>>>>>>>
>>>>>>>>>>>> So from this understanding, it seems that the system should
>>>>>>>>>>>> provide
>>>>>>>>>>>> some
>>>>>>>>>>>> basic pseudo-operating system stuff and programs can build on
>>>>>>>>>>>> that,
>>>>>>>>>>>> just
>>>>>>>>>>>> like they would normally build on their local libc/kernel and
>>>>>>>>>>>> stuff.
>>>>>>>>>>>>
>>>>>>>>>>>> So (I sure like the word "so" today...) if we want the type of
>>>>>>>>>>>>
>>>>>>>>>>> general
>>>>>>>>
>>>>>>>>> os-like stuff it seems their needs to be support for:
>>>>>>>>>>>>   * A simpe message passing model - abstract away all the
>>>>>>>>>>>> TCP-foo,
>>>>>>>>>>>>
>>>>>>>>>>>  maybe
>>>>>>>>>>
>>>>>>>>>>> use existing foo here (obviously needs fleshing out)
>>>>>>>>>>>>   * Permanent storage IO (clone the unix write(), read(),
>>>>>>>>>>>> open()
>>>>>>>>>>>> and
>>>>>>>>>>>> sync()
>>>>>>>>>>>> model,  or maybe just use one of the existing database-ish
>>>>>>>>>>>>
>>>>>>>>>>> nosql
>>>
>>>> things
>>>>>>>>
>>>>>>>>> out
>>>>>>>>>>>> there)
>>>>>>>>>>>>           - Unix-ish model - you create your data hunk, say
>>>>>>>>>>>>
>>>>>>>>>>> you
>>>
>>>> want
>>>>>>>>>>>>
>>>>>>>>>>> all
>>>>>>>>>>
>>>>>>>>>>> this stuff in it, then after sync() we know it's actually
>>>>>>>>>>>> somewhere
>>>>>>>>>>>> written
>>>>>>>>>>>> on a hard-drive, and other things can read it too
>>>>>>>>>>>>           - Unless this isn't in fact needed (but I assume it
>>>>>>>>>>>>
>>>>>>>>>>> is)
>>>>>
>>>>>>           - Also need to figure out if it's filesystem-ish
>>>>>>>>>>>>
>>>>>>>>>>> foo
>>>
>>>> (hierarchial) we want or more relational database-ish stuff
>>>>>>>>>>>>
>>>>>>>>>>>>   * A task delegation model - some type of map/reduce-ish
>>>>>>>>>>>>
>>>>>>>>>>> stuff
>>>
>>>>          - Servers have a few built-in computations, and
>>>>>>>>>>>>
>>>>>>>>>>> client
>>>
>>>> utilizes
>>>>>>>>>>
>>>>>>>>>>> them?
>>>>>>>>>>>>          - Or more complex, servers run sandboxed
>>>>>>>>>>>>
>>>>>>>>>>> computational
>>>
>>>> code?
>>>>>>>>
>>>>>>>>>   * A security system?
>>>>>>>>>>>>        - Needs fleshing out
>>>>>>>>>>>>        - Presumably what the "hub" manages - it's the trusted
>>>>>>>>>>>> thing
>>>>>>>>>>>>        - Obviously, not everybody is allowed to use the
>>>>>>>>>>>>
>>>>>>>>>>> cluster
>>>
>>>> for
>>>>>>>>>>>> computation, not everybody can find out what everybody else is
>>>>>>>>>>>> doing,
>>>>>>>>>>>>
>>>>>>>>>>> etc.
>>>>>>>>>>
>>>>>>>>>>>      - But also, is their a limit on storage, are some things
>>>>>>>>>>>>
>>>>>>>>>>> prioritized
>>>>>>>>>>
>>>>>>>>>>> over others, ?
>>>>>>>>>>>>
>>>>>>>>>>>> Theroretically, server's are written to provide the io backend
>>>>>>>>>>>>
>>>>>>>>>>> and
>>>>>
>>>>>> to
>>>>>>>>>>>> allow
>>>>>>>>>>>> for task delegation, clients use the api, although hub has
>>>>>>>>>>>>
>>>>>>>>>>> it's
>>>
>>>> work
>>>>>>>>>>>> cut
>>>>>>>>>>>> out
>>>>>>>>>>>> delegating all the file io and figuring out what the state of
>>>>>>>>>>>>
>>>>>>>>>>> that
>>>>>
>>>>>> is.
>>>>>>>>
>>>>>>>>> On top of some mixture of this, one could build a simple
>>>>>>>>>>>> unix-ish
>>>>>>>>>>>> pseudo-cli, theroretically, as well as real software.
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway, before actually doing anything, people should read
>>>>>>>>>>>>
>>>>>>>>>>> about
>>>
>>>> PVM
>>>>>>>>>>>> (Parallel Virtual Machine) and the like (maybe also Hadoop and
>>>>>>>>>>>> other
>>>>>>>>>>>> foo-ish
>>>>>>>>>>>> stuff) so Modred isn't just a bad clone of it
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway, (yes, twice in a row!), I figured _someone_ had to
>>>>>>>>>>>>
>>>>>>>>>>> respond
>>>>>
>>>>>> to
>>>>>>>>>>>> Scott,
>>>>>>>>>>>> otherwise he'd feel all lonely and sad :P Now he can have a
>>>>>>>>>>>>
>>>>>>>>>>> warm
>>>
>>>> fuzzy
>>>>>>>>
>>>>>>>>> feeling of deep confusion and uncertainty instead :P
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 25, 2009 at 11:06 PM, Scott Lawrence
>>>>>>>>>>>> <bytbox@xxxxxxxxx>
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>> From: Scott Lawrence <bytbox@xxxxxxxxx>
>>>>>>>>>>>>> Date: Fri, 25 Dec 2009 19:20:13 -0500
>>>>>>>>>>>>> Subject: Design Overview
>>>>>>>>>>>>> To: modred <modred@xxxxxxxxxxxxxxxxxxx>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm going to assume that everyone understands the basic
>>>>>>>>>>>>>
>>>>>>>>>>>> concepts
>>>>>
>>>>>> for
>>>>>>>>
>>>>>>>>> modred: a set of networked computers (by 'networked' I mean,
>>>>>>>>>>>>>
>>>>>>>>>>>> they're
>>>>>>>>
>>>>>>>>> all on the internet), divided for the sake of discussion
>>>>>>>>>>>>>
>>>>>>>>>>>> into
>>>
>>>> three
>>>>>>>>>>>>> classes: the 'hub' (the dude in charge, who compupters who
>>>>>>>>>>>>> want
>>>>>>>>>>>>> to
>>>>>>>>>>>>> join connect to), the 'servers' (dedicated computers that
>>>>>>>>>>>>>
>>>>>>>>>>>> can
>>>
>>>> be
>>>>>
>>>>>> pretty much relied on not to go down, although redundancy is
>>>>>>>>>>>>> always
>>>>>>>>>>>>> nice), and the 'clients' (computers that send in requests
>>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>
>>>> can
>>>>>>>>>>>>>
>>>>>>>>>>>> be
>>>>>>>>
>>>>>>>>> used for spare CPU cycles.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, so much for assumptions... :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Things *I* think any design should emphasize:
>>>>>>>>>>>>>  * security.
>>>>>>>>>>>>>  * relative ease of use, while retaining significant power.
>>>>>>>>>>>>> Challenging.  In particular, it should be possible to set up
>>>>>>>>>>>>>
>>>>>>>>>>>> a
>>>
>>>> modred
>>>>>>>>
>>>>>>>>> network in under an hour, provided the computers are already
>>>>>>>>>>>>>
>>>>>>>>>>>> set
>>>>>
>>>>>> up.
>>>>>>>>
>>>>>>>>>  * along with the previous bullet point, having an interface
>>>>>>>>>>>>> that
>>>>>>>>>>>>> lets
>>>>>>>>>>>>> one use the entire network like a single computer.  This is
>>>>>>>>>>>>>
>>>>>>>>>>>> sort
>>>>>
>>>>>> of
>>>>>>>>>>>>> like the way google docs works, except the cloud is private
>>>>>>>>>>>>>  * therefore, it should be a multi-user system with
>>>>>>>>>>>>> well-designed
>>>>>>>>>>>>> privileges etc...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not going to discuss my implementation ideas, let's hear
>>>>>>>>>>>>> others
>>>>>>>>>>>>> first.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Scott Lawrence
>>>>>>>>>>>>>
>>>>>>>>>>>>> Webmaster
>>>>>>>>>>>>> The Blair Robot Project
>>>>>>>>>>>>> Montgomery Blair High School
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Scott Lawrence
>>>>>>>>>>>>>
>>>>>>>>>>>>> Webmaster
>>>>>>>>>>>>> The Blair Robot Project
>>>>>>>>>>>>> Montgomery Blair High School
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>  Mailing list: https://launchpad.net/~modred
>>>>>>>>>>>>  Post to     : modred@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>>  Unsubscribe : https://launchpad.net/~modred
>>>>>>>>>>>>  More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~modred
>>>>>>>>>>> Post to     : modred@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~modred
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Scott Lawrence
>>>>>>>>>>
>>>>>>>>>> Webmaster
>>>>>>>>>> The Blair Robot Project
>>>>>>>>>> Montgomery Blair High School
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list: https://launchpad.net/~modred
>>>>>>>>>> Post to     : modred@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>> Unsubscribe : https://launchpad.net/~modred
>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Scott Lawrence
>>>>>>>>
>>>>>>>> Webmaster
>>>>>>>> The Blair Robot Project
>>>>>>>> Montgomery Blair High School
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Scott Lawrence
>>>>>>>
>>>>>>> Webmaster
>>>>>>> The Blair Robot Project
>>>>>>> Montgomery Blair High School
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Scott Lawrence
>>>>>>
>>>>>> Webmaster
>>>>>> The Blair Robot Project
>>>>>> Montgomery Blair High School
>>>>>>
>>>>>>
>>>>> --
>>>>> Scott Lawrence
>>>>>
>>>>> Webmaster
>>>>> The Blair Robot Project
>>>>> Montgomery Blair High School
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~modred
>>>>> Post to     : modred@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~modred
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>> --
>>> Scott Lawrence
>>>
>>> Webmaster
>>> The Blair Robot Project
>>> Montgomery Blair High School
>>>
>>>
>>
>> ------------------------------------------------------------------------
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~modred
>> Post to     : modred@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~modred
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>