[Yanel-dev] Canoo WebTest best practices: timing, the 'retry' steap and the 'sleep' task (and he 'timedgroup' step)

[Guillaume Déflache]

Hi!

In some projects we do tests on Javascript code or other kind of 
processing that takes some unpredictable amount of time.
To make sure that these tests have a chance to pass one can:
- need to either wait a bit for the long-running process to complete
- or if the check fails retry the check again immediately
- or do both


Many tests ATM only do one or the other, which is bad because:
- if the check is only tried once and the test machine is too slow at 
that time (say Hudson is compiling another project in parallel which 
hogs the CPU), then the test fails even if the correct result got 
available some time later
- if we keep checking without waiting:
   - we hog the CPU ("busy wait" loop)
   - even with many tries they may be so close from one another they may 
still all fail
   - if we do that many times the Canoo reports get huge which causes 
problems when the report needs to be transformed to HTML


For example *do not do*:

     <sleep description="Wait a bit for GWT JS to initialize" seconds="1"/>
     <verifyText description="Check policy editor name in page body" 
text="Wyona Access Control Policy (GWT) Editor"/>

*neither do*:

     <retry maxcount="100">
       <verifyText description="Check policy editor name in page body" 
text="Wyona Access Control Policy (GWT) Editor"/>
     </retry>

but *rather do*:

     <retry maxcount="10">
       <sleep description="Wait a bit for GWT JS to initialize" 
seconds="1"/>
       <verifyText description="Check policy editor name in page body" 
text="Wyona Access Control Policy (GWT) Editor"/>
     </retry>


Also please do not consider a feature test with no 'retry' and a single 
'sleep' both as a feature test and as a performance test. Better 
explicitely use the 'timedgroup' step 
(<http://webtest.canoo.com/webtest/manual/timedGroup.html>) which is 
designed for that (and probably gives you better reporting WRT execution 
times) in another test, or another performance-testing tool altogether 
like JMeter.


References:
- the full test: 
http://svn.wyona.com/repos/public/yanel/trunk/src/realms/use-cases/yanel/src/test/canoo/tests/usecase-test-policy-manager.xml
- the 'retry' Canoo step (has another example of good usage): 
http://webtest.canoo.com/webtest/manual/retry.html
- the 'sleep' Ant task: http://ant.apache.org/manual/CoreTasks/sleep.html


HTH,
    Guillaume


P.S.: A further improvement to the recommended method above could be to 
make the sleep delay longer after each step, so that we only need a very 
small number of steps to get more and more slack WRT delays.
Cf. the 'decay' parameter in 
http://www.prototypejs.org/api/ajax/periodicalUpdater for a better 
explanation of what I have a mind.
I think this could be made using Canoo's "#{count}" dynamic property, 
but them it would probably be worth factoring that out in a new macro 
then to hide and avoid repeating the gory details.