fema-team team mailing list archive

[John Dunnet <jw.dunnet@xxxxxxxxx>]

Update: I'm pushing what I've got done, but I think it'll certainly help if
I go through it with someone else. See you all tomorrow!
-John


On 13 November 2011 21:51, John Dunnet <jw.dunnet@xxxxxxxxx> wrote:

> Hey guys, I'm having a stab at solving the problem now, so if you are
> trying it, let me know, and we can work at it together. Similarly, if
> someone has already fixed it, let me know.
> -John
>
> On 12 November 2011 08:36, James Newell <jameslnewell@xxxxxxxxx> wrote:
>
>> Hi Guys,
>>
>> We've got a bit of a design flaw at the moment… The way we need the Work
>> Generator and Assimilator to work is:
>>
>> The Work Generator creates the initial work units. When a work unit has
>> been completed BOINC runs the Assimilator. The Assimilator looks up the
>> mapping (jobId, islandId and workunitId) for the work unit being
>> assimilated. Then it needs to get the job and island objects so it can mark
>> the work unit as completed. When the Work Generator finds that enough of
>> the work units are completed it will generate new work units.
>>
>> The issue is that currently in the Work Generator, jobs, islands and work
>> units are stored privately in memory and there is no way that the
>> Assimilator can access the Work Generator's jobs. Even more problematic is
>> that (I assume) the Assimilator and Work Generator are running in different
>> processes on the server and consequently don't have any shared memory.
>> Consequently just putting a getter on the Work Generator won't help at all
>>
>> If someone could put their hand to tackle this issue that would be great.
>> If its not sorted it will mean nothing more than the initial work units get
>> generated and our project isn't that much better than running our problems
>> on our own machine.
>>
>> Firstly confirm that with BOINC the WorkGenerator and Assimilator are run
>> in different processes (pretty certain they are but it would save you a
>> whole heap of work if they're not).
>>
>> Secondly solve the problem!
>>
>> Personally I think that the best way of solving the problem would be to
>> persist data to our database (Jobs, Islands and WorkUnits) so they can be
>> accessed regardless of which process they're created in. If anyone can come
>> up with a hack to get this working before Monday or thinks of a better way
>> please reply all!!!
>>
>> The Assimilator would work something like this:
>>
>>    -
>>    _mapper.getJob(mapping.getJobId()).getIsland(mapping.getIslandId()).getWorkUnit(mapping.getWuId()).markCompleted();
>>
>>
>> Changes to the WorkGenerator would have the following changes:
>>
>>    - move app lookup etc to the mapper
>>    - remove the new jobs loop (the jobs will be pulled from the db for
>>    the process loop and know their state (e.g. new, running, paused, finished,
>>    deleted) and can act accordingly in the process method)
>>
>>
>> Changes to the mapper would be something like:
>>
>>    - Mapper<Factory>(factory, projectPath="/srv/.….") - get the app
>>    name, app id and create the job config (with path handler etc) to pass to
>>    the job
>>    - getJob(id) : Job  - fetch job data from db and construct objects
>>    - getJobs() : vector<Job> - fetch job data from db and construct
>>    objects
>>    - deleteJob(job) - remove job, island and work unit records from DB
>>    and files (in, out, solutions, objectives etc) from disk
>>    - remove getNewJobs() method
>>
>>
>> Changes to the job config would be something like:
>>
>>    - add app name and id
>>
>>
>> Changes to the job would be something like:
>>
>>    - Job(config, id, state, … )
>>    - getIsland(id) : Island - fetch island data from db and construct
>>    objects (could just proxy to the mapper Mapper.getIsland(job, islandId) :
>>    Island)
>>    - getIslands() : vector<Island>  - fetch island data from db and
>>    construct objects  (could just proxy to the mapper Mapper.getIslands(job) :
>>    vector<Island>)
>>    - createIsland() : Island - persist island data in database
>>    - //getBestIndividuals() : Population
>>    - //getBestObjectiveValues() : vector<double>
>>
>>
>> Changes to the island would be something like:
>>
>>    - Island(config, job, id)
>>    - getWorkUnit(id) : Island - fetch work unit data from db and
>>    construct objects (could just proxy to the mapper Mapper.getWorkUnit(job,
>>    island, wuId) : WorkUnit)
>>    - getWorkUnits() : vector<Island>  - fetch work unit data from db and
>>    construct objects  (could just proxy to the mapper Mapper.getWorkUnits(job,
>>    island) : vector<WorkUnit>)
>>    - getSentCount() : int - iterate work units to count
>>    - getCompletedCount() : int - iterate work units to count
>>    - createWorkUnit() : WorkUnit - create boinc work unit, call
>>    create_work() and persist work unit data in database
>>
>>
>> Changes to the work unit would be something like:
>>
>>    - WorkUnit(config, island, wuId, boincId, sent_datetime,
>>    completed_datetime)
>>    - isSent()
>>    - isCompleted()
>>    - markSent() - update database
>>    - markCompleted() - update database
>>    - //getSentIndividuals() : Population
>>    - //getCompletedIndividuals() : Population
>>
>>
>> Yep quite a big job. Please comment.
>>
>> Takers?
>>
>> James
>>
>>
>> --
>> Mailing list: https://launchpad.net/~fema-team
>> Post to     : fema-team@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fema-team
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>