fema-team team mailing list archive

[John Dunnet <jw.dunnet@xxxxxxxxx>]

Hey guys, I'm having a stab at solving the problem now, so if you are
trying it, let me know, and we can work at it together. Similarly, if
someone has already fixed it, let me know.
-John

On 12 November 2011 08:36, James Newell <jameslnewell@xxxxxxxxx> wrote:

> Hi Guys,
>
> We've got a bit of a design flaw at the moment… The way we need the Work
> Generator and Assimilator to work is:
>
> The Work Generator creates the initial work units. When a work unit has
> been completed BOINC runs the Assimilator. The Assimilator looks up the
> mapping (jobId, islandId and workunitId) for the work unit being
> assimilated. Then it needs to get the job and island objects so it can mark
> the work unit as completed. When the Work Generator finds that enough of
> the work units are completed it will generate new work units.
>
> The issue is that currently in the Work Generator, jobs, islands and work
> units are stored privately in memory and there is no way that the
> Assimilator can access the Work Generator's jobs. Even more problematic is
> that (I assume) the Assimilator and Work Generator are running in different
> processes on the server and consequently don't have any shared memory.
> Consequently just putting a getter on the Work Generator won't help at all
>
> If someone could put their hand to tackle this issue that would be great.
> If its not sorted it will mean nothing more than the initial work units get
> generated and our project isn't that much better than running our problems
> on our own machine.
>
> Firstly confirm that with BOINC the WorkGenerator and Assimilator are run
> in different processes (pretty certain they are but it would save you a
> whole heap of work if they're not).
>
> Secondly solve the problem!
>
> Personally I think that the best way of solving the problem would be to
> persist data to our database (Jobs, Islands and WorkUnits) so they can be
> accessed regardless of which process they're created in. If anyone can come
> up with a hack to get this working before Monday or thinks of a better way
> please reply all!!!
>
> The Assimilator would work something like this:
>
>    -
>    _mapper.getJob(mapping.getJobId()).getIsland(mapping.getIslandId()).getWorkUnit(mapping.getWuId()).markCompleted();
>
>
> Changes to the WorkGenerator would have the following changes:
>
>    - move app lookup etc to the mapper
>    - remove the new jobs loop (the jobs will be pulled from the db for
>    the process loop and know their state (e.g. new, running, paused, finished,
>    deleted) and can act accordingly in the process method)
>
>
> Changes to the mapper would be something like:
>
>    - Mapper<Factory>(factory, projectPath="/srv/.….") - get the app name,
>    app id and create the job config (with path handler etc) to pass to the job
>    - getJob(id) : Job  - fetch job data from db and construct objects
>    - getJobs() : vector<Job> - fetch job data from db and construct
>    objects
>    - deleteJob(job) - remove job, island and work unit records from DB
>    and files (in, out, solutions, objectives etc) from disk
>    - remove getNewJobs() method
>
>
> Changes to the job config would be something like:
>
>    - add app name and id
>
>
> Changes to the job would be something like:
>
>    - Job(config, id, state, … )
>    - getIsland(id) : Island - fetch island data from db and construct
>    objects (could just proxy to the mapper Mapper.getIsland(job, islandId) :
>    Island)
>    - getIslands() : vector<Island>  - fetch island data from db and
>    construct objects  (could just proxy to the mapper Mapper.getIslands(job) :
>    vector<Island>)
>    - createIsland() : Island - persist island data in database
>    - //getBestIndividuals() : Population
>    - //getBestObjectiveValues() : vector<double>
>
>
> Changes to the island would be something like:
>
>    - Island(config, job, id)
>    - getWorkUnit(id) : Island - fetch work unit data from db and
>    construct objects (could just proxy to the mapper Mapper.getWorkUnit(job,
>    island, wuId) : WorkUnit)
>    - getWorkUnits() : vector<Island>  - fetch work unit data from db and
>    construct objects  (could just proxy to the mapper Mapper.getWorkUnits(job,
>    island) : vector<WorkUnit>)
>    - getSentCount() : int - iterate work units to count
>    - getCompletedCount() : int - iterate work units to count
>    - createWorkUnit() : WorkUnit - create boinc work unit, call
>    create_work() and persist work unit data in database
>
>
> Changes to the work unit would be something like:
>
>    - WorkUnit(config, island, wuId, boincId, sent_datetime,
>    completed_datetime)
>    - isSent()
>    - isCompleted()
>    - markSent() - update database
>    - markCompleted() - update database
>    - //getSentIndividuals() : Population
>    - //getCompletedIndividuals() : Population
>
>
> Yep quite a big job. Please comment.
>
> Takers?
>
> James
>
>
> --
> Mailing list: https://launchpad.net/~fema-team
> Post to     : fema-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fema-team
> More help   : https://help.launchpad.net/ListHelp
>
>