modred team mailing list archive

[Scott Lawrence <bytbox@xxxxxxxxx>]

My idea here, as I explained to Mikey last night over chat, is that
the transparent access allows us to run programs out-of-the-box. Most
programs won't perform well like this, for various reasons, and that's
fine, a few will perform ok, and a few might happen to perform
excellently.  But these programs could be tweaked by telling the
cluster that when the program forks in such-and-such place, it should
be put on another computer, and the program should generally be
packaged with the following data, etc... etc..., and stuff like that
to minimize network usage.

Programs written for the cluster will generally not want to use the
transparent features.  Of course, if they do, it'll still work.


On 12/30/09, Michael Cohen <gnurdux@xxxxxxxxx> wrote:
> If you want networked memory access, you need to be using mmap()
> somewhere.  I agree with you that it is probably not such a great idea;
> Scott wants transparent stuff over the cluster but in reality that is
> probably pretty much not a possibility, as there is a big difference
> between .0000001 second latency and .1 second latency, so any code that
> makes almost any use of any inter-process communication would run many
> times slower on the cluster than on one computer.
>
> Michael Cohen
>
> Frederic Koehler wrote:
>> Uh...what exactly is this supposed to be used for?
>>
>> Trying to abstract networked data as mmap over some type of network file
>> system
>> is an ok option, but it's not simple (for one, figuring out the
>> synchronization and the like, and also having to build a file system or
>> finding some obscure one which meets requirements).
>>
>> Also, it's not easily usable for clients, so if they need to access data
>> they still probably need some more traditional interface.
>>
>> On the other hand, a more traditional interface (for ex. cloning read(),
>> write() and the like in a user-space library, or some other approach)
>> could
>> be used for both.
>>
>> Anyway, though, doing lots of reading/writing over the network is really
>> bad
>> performance-wise (it clogs up the network and has horrible latency), so I
>> don't know if whatever you're planning is actually a good idea (especially
>> if "memory access often involves the cluster"...).
>>
>>
>> On Wed, Dec 30, 2009 at 12:01 AM, Michael Cohen <gnurdux@xxxxxxxxx> wrote:
>>
>>> I just realized that we could actually do this by mmapping to stuff in
>>> some
>>> kind of network file system.  It might be a little tricky to ensure that
>>> the
>>> data is actually up to date, but it probably could be done.
>>>
>>> Michael Cohen
>>>
>>>
>>> Scott Lawrence wrote:
>>>
>>>> I, the perennial linux newbie, was somehow under the impression that
>>>> memory access was a system call.  Ouch.
>>>>
>>>> So the issue is that memory access often involves the cluster, and so
>>>> we should have a fast way to interrupt it to talk over the network.
>>>> (Obviously, we wouldn't re-fetch the same data repeatedly.)
>>>>
>>>> Mikey suggested mmap.  The issue is that this only maps to actual
>>>> files, so it wouldn't work for network access.  Anyone know of a way
>>>> to mount a kernel driver in user space?  Or is there a better way?
>>>>
>>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~modred
>>> Post to     : modred@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~modred
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~modred
> Post to     : modred@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~modred
> More help   : https://help.launchpad.net/ListHelp
>


-- 
Scott Lawrence

Webmaster
The Blair Robot Project
Montgomery Blair High School