mvhub-dev team mailing list archive
-
mvhub-dev team
-
Mailing list archive
-
Message #00543
91.350 class notes #3 (future)
############ Stuff to type when you sit down to work #################
ssh brave.thecsl.org
# move to /var/www/mvhub/$USER/source-code
cdws
bzr branch lp:mvhub
# or:
cd trunk && bzr pull
# see what's been merged lately
bzr log | more
# see commit messages for
# merged branches
bzr log --include-merges | more
# see commit messages for
# merged branches
# AND include filenames
bzr log -v --include-merges | more
# see the diff each merged
# made to trunk
bzr log --show-diff | more
# randomly see who is to blame.
trunk::bzr blame lib-mvhub/lib/MVHub/Utils/Setup.pm | more
# randomly see who is to praise
bzr praise app-mvhub/setup/database/sql/009_drop_get_agency_name.sql| more
# create a branch to make your test
# speedup
# don't use 'filename' literally
bzr branch lp:mvhub speedup_filename_t
# pick your branch
mv_set active
# update the rest of your development enviroment
mv_update_development
##############
Right now tests take more than 2+ minutes to run, (this is better than
the 5 minutes they used to take). Tests should run in less than 30
seconds, if we want people to run them regularly.
The way we do tests should change so as we add more tests to get better
test coverage, it doesn't take significantly longer to run our tests. to
close this bug, adding more tests should increase test run time n(log n)
(if I remember my school boy asymptotics correctly)
In approximate order of priority, approaches to fixing this problem are:
1) identify hotspots (time consuming test files)
a. time each test file
<*****above is most important*****>
2) adopt best strategy to speed up slow file
a. Cache test results
a-1. Do some profiling / identify hotspots in each file
b. run tests in parallel
c. Run tests as part of a persistent process.
d. Aggregate some tests to save process & repeated compile overhead
./lib-mvhub/t/03-run_test_class.t vs lib/t/Utils/Setup/*.t
e. Speed up external dependencies (webserver , network)
##########
Identify hotspots
##########
From the fine manual:
man prove
...we know that
prove --timer -r <files or directories> > raw_test_times.txt
...will give us the amount of time in milliseconds each test file takes
to run.
from our knowledge of shell and 'man sort'
we know:
# sort in numeric (not alpha) order, reverse on 5th column
# redirect output to file sorted.csv
sort -nrk 5 raw_times.txt > sorted.csv
Because we're not command line snobs, we can use open office spreadsheet
to do some
summing and dividing and see if the 80/20 rule works here.
scp
brave.thecsl.org:/var/www/mvhub/omacneil/source-code/trunk/sorted.csv .
oocalc sorted.csv
Surprise, surprise our top 20 (of 98) time consuming test scripts,
consume 84% of the test time. (123 seconds of 145 of total runtime)
Test scripts we should optimize for best results are:
app-mvhub/t/mech/pages.t ..................................... 66684
app-mvhub/t/mech/user/login_with_bad_password.t .............. 9041
app-mvhub/t/mech/admin/01_add_complete_agency.t .............. 7512
app-mvhub/t/mech/user/stuff_search_contact.t ................. 6427
app-mvhub/t/bugs/guide.pl_incomplete_path.t .................. 6181
app-mvhub/t/check_syntax.t ................................... 5551
app-mvhub/t/mech/user/contact_missing_required.t ............. 2736
app-mvhub/t/mech/user/login_with_good_password.t ............. 2151
app-mvhub/t/bugs/ql_display_agency_home_interaction.t ........ 2054
app-mvhub/t/bugs/categorize_link_missing_when_agency_login.t . 1933
app-mvhub/t/bugs/ql_works_after_admin_login.t ................ 1315
app-mvhub/t/mech/user/agency_search_form_found_mvh_only.t .... 1194
app-mvhub/t/mech/user/category_search_form_found_in_all_db.t . 1180
app-mvhub/t/mech/user/agency_search_form_not_found.t ......... 1177
app-mvhub/t/mech/user/category_search_form_not_found.t ....... 1138
app-mvhub/t/mech/user/agency_search_form_found_nsp_only.t .... 1134
app-mvhub/t/mech/user/contact_having_required.t .............. 1110
app-mvhub/t/mech/user/agency_search_form_blank.t ............. 1089
app-mvhub/t/mech/user/category_search_form_blank.t ........... 1088
lib-mvhub/t/03-run_test_class.t .............................. 1065
lib-mvhub/t/00-setup.t ....................................... 934
In case it isn't clear, reducing runtime by 99.9% for
lib-mvhub/t/Utils/assert.t isn't going to do much for us since it takes
which takes about 1/10 of a second to run.
#############
a. Cache test results
#############
Meta tests that run on a bunch of files are good candidates for this.
See:
perldoc TestHelper::History
# only 24 lines !
more app-mvhub/t/check_syntax.t
See also: <your files here>
# maybe some test files that work on 'files' name data structures after
'files'
grep '\@.*files' . -lr | grep 't$'
# maybe these test files are in the top 20 (or not)
############
a-1. Do some profiling / identify hotspots in each file
############
# See if anything jumps out in our top 20 list
# some tests require some setup
lib-mvhub/t/00-setup.t && lib-mvhub/t/01-load.t && mv_profile
lib-mvhub/t/03-run_test_class.t
firefox http://nsp.$USER.testing123.net
# this isn't on our top 20 list any more
# at least for re-runs, but the backticks are interesting
mv_profile app-mvhub/t/perlitdy.t
############
b. explore running tests in parallel
#############
cdws
bzr branch lp:~omacneil/mvhub/tap_harness_play
mv_set_active
cdw
# prelimiary results not so promising
time mv_prove_all
time mv_prove lib-mvhub/t
# breaking appt-mvhub/t/mech/pages.t into seperate files
# that can be run simultaniously may help
# It may be we don't get a big win on the one core brave.thecsl.org has
available.
###########
Skip this until you learn more Perl
Basically Test::Class
c. Run tests as part of a persistent process / Aggrigate .
d. Aggregate some tests to save process & repeated compile overhead
./lib-mvhub/t/03-run_test_class.t vs lib/t/Utils/Setup/*.t
###########
e. Speed up external dependencies (webserver , network)
###########
probably requires root and some big sys-admin work
see mod_perl:
