My idea here, as I explained to Mikey last night over chat, is that
the transparent access allows us to run programs out-of-the-box. Most
programs won't perform well like this, for various reasons, and that's
fine, a few will perform ok, and a few might happen to perform
excellently. But these programs could be tweaked by telling the
cluster that when the program forks in such-and-such place, it should
be put on another computer, and the program should generally be
packaged with the following data, etc... etc..., and stuff like that
to minimize network usage.
Programs written for the cluster will generally not want to use the
transparent features. Of course, if they do, it'll still work.
On 12/30/09, Michael Cohen <gnurdux@xxxxxxxxx> wrote:
If you want networked memory access, you need to be using mmap()
somewhere. I agree with you that it is probably not such a great idea;
Scott wants transparent stuff over the cluster but in reality that is
probably pretty much not a possibility, as there is a big difference
between .0000001 second latency and .1 second latency, so any code that
makes almost any use of any inter-process communication would run many
times slower on the cluster than on one computer.
Michael Cohen
Frederic Koehler wrote:
Uh...what exactly is this supposed to be used for?
Trying to abstract networked data as mmap over some type of network file
system
is an ok option, but it's not simple (for one, figuring out the
synchronization and the like, and also having to build a file system or
finding some obscure one which meets requirements).
Also, it's not easily usable for clients, so if they need to access data
they still probably need some more traditional interface.
On the other hand, a more traditional interface (for ex. cloning read(),
write() and the like in a user-space library, or some other approach)
could
be used for both.
Anyway, though, doing lots of reading/writing over the network is really
bad
performance-wise (it clogs up the network and has horrible latency), so I
don't know if whatever you're planning is actually a good idea (especially
if "memory access often involves the cluster"...).
On Wed, Dec 30, 2009 at 12:01 AM, Michael Cohen <gnurdux@xxxxxxxxx> wrote:
I just realized that we could actually do this by mmapping to stuff in
some
kind of network file system. It might be a little tricky to ensure that
the
data is actually up to date, but it probably could be done.
Michael Cohen
Scott Lawrence wrote:
I, the perennial linux newbie, was somehow under the impression that
memory access was a system call. Ouch.
So the issue is that memory access often involves the cluster, and so
we should have a fast way to interrupt it to talk over the network.
(Obviously, we wouldn't re-fetch the same data repeatedly.)
Mikey suggested mmap. The issue is that this only maps to actual
files, so it wouldn't work for network access. Anyone know of a way
to mount a kernel driver in user space? Or is there a better way?
_______________________________________________
Mailing list: https://launchpad.net/~modred
Post to : modred@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~modred
More help : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~modred
Post to : modred@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~modred
More help : https://help.launchpad.net/ListHelp
