Living Up to Expectations

University of California, Irvine

Meeting the Requirements and Living Up to Expectations

Kristina Winbladh awinblad@ics.uci.edu

University of California, Irvine

Rand Waltzman

Royal Institute of Technology

rand@nada.kth.se

Thomas A. Alspaugh alspaugh@ics.uci.edu

University of California, Irvine

Debra J. Richardson djr@ics.uci.edu

University of California, Irvine

January 2007

ISR Technical Report # UCI-ISR-07-1

Institute for Software Research

ICS2 110

University of California, Irvine

Irvine, CA 92697-3455

www.isr.uci.edu

www.isr.uci.edu/tech-reports.html

MeetingtheRequirementsandLivingUptoExpectations

KristinaWinbladh,ThomasA.Alspaugh,DebraJ.Richardson

InstituteforSoftwareResearchDepartmentofInformatics

DonaldBrenSchoolofInformationandComputerSciences

UniversityofCalifornia,Irvine{awinblad,alspaugh,djr}@ics.uci.edu

RandWaltzman

DepartmentofComputerScienceRoyalInstituteofTechnology

rand@nada.kth.seISRTechnicalReportUCI-ISR-07-1

January30,2007

Abstract:Toproducebetterqualitysoftwareatreasonablecost,weproposerequirements-basedtesting,inwhichtestingisdrivendirectlyfromtherequire-mentsandfaultsthatpreventtheproductfrommeetingitsrequirementsaredetected.Ourapproachmakesuseofrequirementsintheformofgoalsandscenarios.Fromthesewegeneratetestscenariosthatdrivethesystemundertestthroughparticularpathsofthescenarios,andatestharnessthatverifiesthesystemfollowstheparticularpathandmeetsitsconditions.Becauseourtestscenariosarederiveddirectlyfromtherequirements,amajorbenefitoftheprocessofwritingtestscenariosistheidentificationofpoorlyformulatedrequirements.WeappliedourapproachtoasamplesoftwaresystemandtomutantsofitgeneratedbyMuJava.Ourapproachwaseffectiveatfindingim-plementationfaultsthatcausedthesystemtodivergefromtherequirements.

1

MeetingtheRequirementsandLivingUptoExpectations

KristinaWinbladh,ThomasA.Alspaugh,DebraJ.Richardson

InstituteforSoftwareResearchDepartmentofInformatics

DonaldBrenSchoolofInformationandComputerSciences

UniversityofCalifornia,Irvine{awinblad,alspaugh,djr}@ics.uci.edu

RandWaltzman

DepartmentofComputerScienceRoyalInstituteofTechnology

rand@nada.kth.seISRTechnicalReportUCI-ISR-07-1

January30,2007

Abstract

Toproducebetterqualitysoftwareatreasonablecost,weproposerequirements-basedtesting,inwhichtestingisdrivendirectlyfromtherequirementsandfaultsthatpreventtheproductfrommeetingitsre-quirementsaredetected.Ourapproachmakesuseofrequirementsintheformofgoalsandscenarios.Fromthesewegeneratetestscenariosthatdrivethesystemundertestthroughparticularpathsofthescenarios,andatestharnessthatverifiesthesystemfollowstheparticularpathandmeetsitsconditions.Becauseourtestscenariosarederiveddirectlyfromtherequirements,amajorbenefitoftheprocessofwritingtestscenariosistheidentificationofpoorlyformulatedrequirements.WeappliedourapproachtoasamplesoftwaresystemandtomutantsofitgeneratedbyMuJava.Ourapproachwaseffectiveatfindingimplementationfaultsthatcausedthesystemtodivergefromtherequirements.

1Introduction

Thegoalofsoftwaretestingistoassessthequalityofasoftwareproductbyfindingfaultsinit.Theeffectivenessandefficiencyofatestingapproachcanbecharacterizedintermsoftradeoffsbetweentheeffortofusingtheapproachanditsabilitytofindfaults.Specification-basedtestingiseffectiveandefficient

1

inthesensethattestingresourcesarefocusedonthemostimportantbehav-iorofthesystemandinadditiontocodefaultsitcanalsorevealfaultsinthespecificationearly.Whenspecificationfaultsarerevealedearly,wecanpreventthesefrompropagatingintothefinalproduct.Webelievethatrequirements-basedtestingisaparticularlyeffectivespecification-basedtestingapproach.Inrequirements-basedtesting,therequirementsserveasthebasisof(1)theval-idationofthesystemundertestand(2)thescenariosthatdrivethesystemwhileitistested.Inadditiontothebenefitsofspecification-basedtesting,requirements-basedtestingtestsagainsttherequirementsmostimportanttostakeholdersandassessesqualityofthesystemundertestintermsstakeholdersunderstand.

Theobjectiveofourworkistoutilizethenaturalsymbioticrelationbetweenrequirementsandtestingtoproducesystemsthatbettersatisfystakeholderneeds.Wedothisbyusingthetestingprocesstovalidatetherequirements,andusingtestresultstodemonstratethequalityofthesoftwareintheextentthatitmeetstherequirements.TherelationisdepictedinFig.1asathree-wayexchangeofbenefits.Thefirstbenefitisthatresultsoftestsgeneratedfromtherequirementsprovidemeaningfulinsightsaboutthequalityofthesystemundertest,intermsofhowwellitmeetsitsrequirements.Thesecondbenefitisthatearlytestdesignhelpstobuildqualityintotheproductbyvalidatingtherequirementsandinhibitingdefectmultiplication.Testingagainstaspec-ificationgeneratesquestionsaboutthatspecificationandchallengesit.Iftestdesignisconsideredearlyanddrivenbyrequirements,therequirementsreceiveadditionalvalidationbeforerequirementsproblemscancausecostlymisunder-standingslaterinthedevelopmentprocess[7].Thethirdbenefitisthattestsgeneratedfromrequirements,asopposedtolaterspecificationsandmodelsofcode,offersubstantialopportunitiesfortestingefficiencyandallowthegener-atedtestcasestobedirectlytracedbacktohigh-levelrequirements[17].

generate meaningful testsRequirementsvalidate requirementstraceabilityTestingFigure1:Symbioticrelationbetweenrequirementsandtesting

Inthispaperwewilldescribearequirements-basedtestingapproachinwhichtestsaredirectlydrivenbyandcheckedagainstrequirementsintheformofsce-narios.Weappliedourapproachtoasamplesystem,AquaLush[6],aprojectdevelopedelsewherewithafullrangeofartifacts.AquaLushisanautomaticirrigationsystemthatcontrolsirrigationbasedonsoilmoisturelevelsratherthantiming.WewilluseAquaLushartifactsandconceptstodemonstrateourapproachthroughoutthepaperandvalidateourapproach.Insection2,wesummarizerelatedwork,andinsection3wedescribetherequirementsspecifi-cationformatthatourapproachuses.Wepresentthedetailsofourapproachinsection4.Insection5wedescribetheresultsofourvalidationstudyandwe

2

concludethepaperwithlessonslearnedandfutureworkinsection6.

2RelatedWork

Alltoooften,testingiscutshortwhenbudgetsorresourcesareshort,doesnotcorrespondtotheoriginalrequirements,andisgenerallyinadequate,ineffi-cient,andineffective.Softwareengineeringresearchershavelongwarnedofandpointedouttheenormousrisksassociatedwithneglectingsoftwaretesting[15].Prudentcostmanagementthereforerequiresthatsoftwaretestingalsobeeffi-cient.Testingresearchhasfocusedmainlyoncode-basedtesting,inwhichtestsaredevelopedandchoseninordertoachievecoverageoftheimplementationcode.Althoughcode-basedtestingcansuccessfullydetectfaultsinthecode,itmightnotdetectfaultsthatproducebehaviorthatisplausiblebutfailstomeetthesystem’srequirements.

Specification-basedtesting,ontheotherhand,isatestingtechniquewhosepurposeistoconfirmtheextenttowhichasystemunderdevelopmentmeetsitsspecifications.Whereascode-basedtestingstrivestoexerciseasmuchoftheimplementationaspossibleinordertorevealfaultsintheimplementa-tion,specification-basedtestingstrivestoexaminewhethertheimplementationmeetsasmuchofitsspecificationaspossible,inordertorevealfaultsthatpreventtheimplementationfromdoingso.Mostspecification-basedtestingap-proacheshavefocusedonlower-levelspecificationsthataretypicallyexpressedintheformofLabeledTransitionSystems,FiniteStateMachines(FSM),statecharts,ormessagesequencecharts(dependingontheapproach).Itispossibletoautomaticallygeneratetestsuitesfromthesethatcandeterminehowwellthesystemconformstoitsspecification[11].Sincehigh-levelrequirementstypicallyarelessformalandmoreabstractthancomponentspecifications,specification-basedtestinghasnotbeensuccessfullyappliedtothembeforenow.

Therehasbeenworkontestcasegenerationfromrequirements,particu-larlyfromUMLusecases[4,12].Theseapproacheshowever,seemtodependondesigninformation.Ourapproachdiffersfromtheseinthatitispurelyrequirements-based,i.e.wedonotmakeuseofdesignorimplementationinfor-mationwhencreatingtestcasesortestoracles.Morethanonesetofdesignchoicesandimplementationscanthereforebevalid,andbetestedagainstonesetofrequirements.Althoughhigh-levelgoalsandscenariosarenotasformalasFSMs,webelievethattestingagainstrequirementsexpressedinthesetermsad-dressesmanyofthecommonlyrecognizedproblemsincreatingqualitysoftwareunderacceptablebudgetandtimeconstraints.

Themechanismtoevaluatewhethertheoutputofatestscenarioiscorrectisknownasatestoracle,andthebeliefthatthetesterisabletodeterminewhetherthetestoutputiscorrectisknownastheoracleassumption[16].Atestoracleconsistsoftwomainparts:(1)expectedoutputfromthesystemundertest,and(2)aprocedurethatcomparestheexpectedoutputwiththeactualoutput[13].Oraclescaneitherbehuman(i.e.,manualcheckingofoutput)orautomated(e.g.,software),andalthoughtheyseemessentialtotesting,they

3

areoftennoteasytocomeby.Forallintentsandpurposes,softwareoraclesdonotexistincommonindustrialpractice,whilethequalityofhumanoraclesandtheirtimeandeffortisalmostnevertakenintoaccountintheevaluationoftestingmethodseventhoughhumantestersarefrequentlyunsureofthecorrectnessoftestoutputandmustrepeattheirworkeverytimethetestsarerun.Thisindicates,asrecentworkalsoshows,thattestoraclescanhaveasignificantimpactontesteffectivenessandefficiency[20].Weproposeamoreautomatedapproachinwhichwetestthesystemagainsttestoraclesderivedfromrequirements,sothatresultscanbeautomaticallyandreliablychecked.Inpriorworkwemanuallyconstructedtheseoraclesfromthepre-andpost-conditionsofgoalsintheplans,sothattheoraclescouldanswerwhetherthesystem’sstatechangedfromagoal’spre-conditiontoitspost-condition.Sincepre-andpost-conditionsareexpressedformallyinpredicatelogic,thereisgoodreasontobelievethattestoraclesforthemcanbeautomaticallygenerated.

Workonrelationshipsbetweengoalsandrefinementofgoalsintoopera-tionalizablerequirementshasbeencarriedoutforatleastadecade.Thisworkincludesthereductionofgoalstofunctionalandnon-functionalrequirements.OfparticularinterestistheworkongoalrefinementbyLamsweerdeetal.[9],Mylopoulos,Chung,Yu,etal.[5],andRollandetal.[14],specificallytheworkonmappinggoalstorequirementsscenarios.

Preliminaryresultsofourpreviousworkindicatedthattestingagainstgoalsandplanscansuccessfullydistinguishfalsepositivetestresultsanddomainknowledgeerrors[19].Ourpreviousworkalsoindicatedthatwecaninfertheachievementofhigh-levelstakeholdergoalsfromtheachievementofgoalsattheimplementationlevel[18].WefoundevidencethatstakeholderspreferSce-narioMLscenariosoverusecasesandmessagesequencecharts[2].Althoughourpreliminaryresultsarepromising,wedetectedsomedrawbacksusinggoal-annotationsinthesourcecodeandnotprovidingsupportfortestgeneration.Wenowfeelencouragedtoexplorethedevelopmentofarequirements-basedtestingframeworkthataddressestheseissues.

3GoalsandScenarios

Goalsdescribestakeholders’intentionsforthesystemandstateobjectivesthatthesystemshouldmeet.High-levelstakeholdergoalsarestep-wiserefinedinanAND/ORgoal-graphintolower-levelfunctionalgoals.Atsomepointintherefinementprocess,theorderofsub-goalscanbecomerelevant.Wethereforeintroducetheconceptofplans.Aplanisanabstractdescriptionofhowtosatisfysomegoalbysatisfyingitssub-goalsinsomesequence.WeuseGoalML,anXML-basedlanguage,toexpressgoalsandplans[19,18].XML-basedlanguagesareparticularlyusefulforsupportingtestautomationandproductionofhumanreadableversionsofscenariosandgoalmodels.

Atanevenlowerlevelofabstraction,goalscanberefinedbyscenarios.Ascenariodescribesauseofasystemintermsofsituations,interactionsbe-tweenagents,andeventsunfoldingovertime.ScenarioMLisanXMLlan-4

guageforscenarios[2,3].ScenarioMLexpressesscenarioswithacombinationofevents,ontologies,references,andscenarioparameters.Theeventsarere-cursivelystructured,startingfromsimpletexteventsasabasis,andincludingcompoundeventsgroupingseveraleventsinaparticularorder(totalorpartial),eventschemassuchasiteratedeventsandsetsofalternativeevents,andepisodesthatspecifyanotherscenarioasanevent.Allen’sintervalalgebrarelations[1]expressthetemporalrelationshipsamongthepartsofacompoundevent.Thisapproachtoeventssupportsautomatedrecognitionofscenarioshappeninginadomain,derivationofonescenariofromanother(suchasoneormoretestscenarios,orpathsthrougharequirementsscenario),andotherautomatedpro-cessing.Ontologiesgiveawaytodescribethekindsofentitiesthatcanexistinadomain,definespecificentities,andexpresstherelationshipsamongthem.Weuseontologiestohelpgivethecontextofscenariosand(throughscenarioparameters)specifytherangeofentitiesthatcanappearinaspecificscenario.Wehaveseenthatwithoutontologiesandscenarioparameters,itisdifficulttoderiveadequatetestsfromrequirementsscenariosbecausethereisnoinfor-mationwithwhichtomakethemconcrete,andthereislittleopportunityforautomationoftheprocess.

Figure2showsapartialgoalgraph.Thefigureillustratestherefinementofatop-levelstakeholdergoal“Createahigh-qualitymaintainableproduct”throughvariouslowerlevelgoalssuchas“AquaLushmustirrigateonlyuntilthecriticalmoisturelevelisachieved”and“AquaLushmustallowoperationineithermanualorautomaticmode”downtothe“IrrigateScenario”.The“IrrigateScenario”isarefinementofthegoals,becauseifthesystemdoeswhatthescenariospecifies,thereisconfidencethatthecorrespondingstakeholdergoalsaremet.Figure3showsasmallsnippetofthe“IrrigateScenario”.Thescenarioshowsthatthesystemshouldtrytoreadeachsensorthreetimes,ifitfailsafterthreetimesthesensorismarkedasfailed.Thenextsequenceinthescenario(Sequence2)hasapre-conditionbasedontheresultofthesensorreadsinthetopofthescenario.

4Requirements-BasedTesting

Testingistypicallybedividedintofourmajoractivities:testdesign,testgen-eration,testexecution,andtestevaluation.InthefollowingsectionswewilldescribeourtestingapproachthroughtheseactivitiesandillustrateitwitharunningexamplefromAquaLush.

5

Create a high-quality maintainable productAquaLush must allow It is quick and easy to operation in either tell when the product It is quick and easy to It is quick and easy to manual or automatic is not working track down problemsfix problemsmodeproperlyDef: manual, automaticManual:AutomaticAquaLush must report on whether the system has failed componentsAquaLush must report what is wrong with the system when it does not workAquaLush must irrigate only during irrigation timesAquaLush must irrigate only until the set critical moisture level is achievedAquaLush must irrigate only until the water allocation is achievedAquaLush must report all persistent store failuresAquaLush must be able to detect valve and sensor failuresIrrigation must occur at specified timesIrrigateScenarioFigure2:Partialgoal-graph

4.1TestDesign

Webelievethetestingprocessshouldbeginasearlyaspossible.Wethereforesuggestthattestdesignshouldbeginassoonastherearerequirements.Fig-ure4showsadiagramofthetestdesigntasksinourapproach.Thegoalofourtestingapproachistodrivethesystemthroughavarietyofpathsthroughthescenarios,compareeventsoutputfromthesystemtoeventsintherequire-mentsscenario,andevaluatewhetherornottheymatch.Theeventscanbedividedintothreekindsofevents:internal,boundary,andexternal.Internaleventsareeventsinsidethesystemundertest,andarenotcheckablethroughspecification-basedtesting.Anexampleofsuchaneventis“AquaLushplacesthezoneonanactivezonelist”.ThiseventisinternaltotheAquaLushsystem,andcannotbedetectedbymonitoringthesystemfromtheoutside.Internaleventsaretypicallynotrequirements-leveleventsbecausetheydealwithdesignlevelissues.Boundaryeventsareeventsthatinvolveboththesystemundertestandthetestingenvironment.Anexampleofsuchaneventis“AquaLushfailedtoreadthesensor”.Thiseventisanoutputfromthesystemundertesttoanexternalfunction,whichmeansthatwecanmonitortheoccurrenceoftheeventfromoutsidethesystemundertest.Mostoftheeventsinourre-quirementsscenariosareofthistype.Externaleventsareeventsthathappen

6

Figure3:Scenariosnippet

outsidethesystemundertest.Anexampleofsuchaneventis“Itisraining”.ThisisaneventthatdoesimpactthemoisturelevelofthesoilthatAquaLushirrigates.AlthoughwedonothaveanyrequirementsscenariosthatcontainexternaleventsinAquaLush,wecandesignteststosimulatesuchoccurrencesandoraclestoverifycorrectsystembehaviorundersuchcircumstances.

Fromeachrequirementsscenariowemapthescenarioeventstoinputandoutputfunctionsthatwillconnectthesystemundertestandthetestharness.Wethenusetherequirementsscenarioandthemappingtocreateseveraltestscenarios,witheachtestscenariotracingaparticularpaththroughtherequire-mentsscenario.Infigure4thiscorrespondstofollowingthetoparrow‘generate’fromgoals,plans,andscenariostotestscenario.Forexample,inthescenariosnippetoffigure3,thealternativespresentfivedifferentpaths.However,sincetheiterationspecifiesforeachsensor,fivesensorsallowonetestscenariotocoverallpaths,threetofoursensorsallowtwotestscenarios,andsoforth.

Whenmappingtherequirementsscenarioeventstotestharnessfunctions,welookattheeventsforsystemoutputandinput.Therequirementsscenarioevent“Failedtoreadsensor”,forexample,isdividedintoasystemoutputpart,whichthetestharnessshouldbemonitoringoccurrencesof,andasysteminputpart.Thesysteminputpartistheresultoftheevent,orinputto“drive”thesystemincaseswherethisisneeded.Inourexample,thesystemoutputis“Readsensor”,andmapstotestharnessfunctionSimSensorDevice.read(),andthesysteminputis“fail”,whichmapstothereturnstatementofthetestharnessfunctionSimSensorDevice.read()

Theprocessofmappingscenarioeventstoinputandoutputfunctionscanrevealwhetherornottheeventsinthescenarioaretestable.Insection5wewilldescribeinstancesofnon-testableevents(internalevents)thatwediscoveredinAquaLush’susecases.Sinceeacheventisevaluatedfortestability,thisprocess

7

generateTest Scenariosdefine test harness API used to constructused to constructused to constructinstantiate tests with datavalidateTest Harness APIdefine return functionsOracleScenario RecognizerDriverTest CaseTest HarnessFigure4:TestDesignFlow

servesasonevalidationoftherequirements.

ThemappingofscenarioeventstosysteminputandoutputallowsustodefinethetestharnessAPI(suchasthefunctionsmentionedabove),whichthesystemcanhookintolater.Asimulatedenvironmentisoftenneededwhentestingsoftwaresystems.AsimulatedenvironmentforAquaLushwasoneoftheartifactsprovideby[6],butformostprojectssuchanenvironmentwouldhavetobebuilt.WemadeAquaLush’ssimulationapartofthetestharnessbyalteringittoprovidethecapabilityofmonitoringcallsandprovidinginput.TheSimSensorDeviceclassforexample,containsinstancesoftheSensorDeviceclasswhichAquaLushwillcommunicatewithwhenreadingmoisturelevelsfromitssensors.AlthoughwecannotaddanymonitoringcodetoSensorDevice,becauseitispartofthesystemundertest,weareabletomonitorthecallstothesensorsthroughSimSensorDeviceinstead.

Figure5showsaruntimeviewofthetestharness.Oncethedifferentcom-ponentsareconstructedandintegrated,eventsflowfromthesystemintothetestharnesswheretheyarematchedagainsttheexpectedeventsgivenbythecurrenttestcase.Whenaneventismatchedagainstthetestcase,thetestcaseprovidesthenextinputneededtodrivethesystemalongthechosenpath.Theoraclesusetheruntimeinformationalongwithrequirementsknowledgetocom-putecorrectresults,andtheruntimepre-andpost-conditionsinthescenarioarecheckedagainsttheseresultsforcorrectness.

8

Test Casedecide outcomeDrivermatch andmove onprovide inputto the systemruntime monitoringof eventsScenario RecognizerTest Harness APIOraclesSystemChecks whetherChecks whether event scenario is followedconditions are metFigure5:RuntimeFlow

Duringtestdesignwecanalsoconstructtherestofthetestharnesssinceitisindependentofthedevelopmentofthesystemundertest.Thescenariorecognizermatcheseventscomingintothetestharnessfromthesystemundertesttoexpectedeventsinthetestcase.Thescenariorecognizerisconstructedusingthestructureandinformationintherequirementsscenario.Eachtestcaseisapieceofcodethatcontainsanorderedlistofevent-responses.Theseevent-responsescontaininformationaboutthetypeofevent,particulareventparameters,andaparticularresponsetotheevent.Thetestcasesarespe-cializationsofthetestscenarioscreatedbyaddingconcreteinputdataforthesystemundertest.Thedriverisapieceofcodethatusestheinformationfromtheevent-responsetodrivethesystembyprovidingitinput.ThedriverisconstructedbyusinginformationintherequirementsscenariosandthetestharnessAPI.Theoraclesarepiecesofcodethatcomputeexpectedresultsbasedonthetestcaseandtestcaseinformationavailableatruntime,andcomparesthistoactualresultsoftherunningsystem.Weconstructtheoraclesmanuallyusingtheinformationintherequirementsscenarios.Foroursamplesystem,weconstructedthetestharnessAPIfromtheAquaLushsimulationinJava,andimplementedthescenariorecognizer,driver,testcases,andoraclesintherule-basedlanguageJESS(JavaExpertSystemShell)[8].

Figure6showsasampleJESSruleforthedriver.Thelefthandsideoftherulestatesthattherehastobeataskforactivatingazoneandanevent

9

forreadingasensorinthatzonethatwasmatchedbythescenariorecognizer;thenumberoftimesthissensorhasbeenattemptedtoreadmustbebelowthemaximumnumberallowed(threeaccordingtotherequirementsscenario);andthereexistsanevent-responseinthetestcasefortheeventandthezonewhichhasnotyetbeenprocessed,andwhoseoutcomehasthevalueof‘fail’.Therighthandsideoftherulestatesthatifthelefthandsideoftheruleistrue,thenthenumberoftimesthissensorwasreadwillbeincrementedbyone,theresultofthesensorreadwillbesetto-1,thestatusoftheeventwillbesettoprocessed,andthestatusoftheevent-responsewillbesetto‘done’.

(defrule process-read-sensor-request-2

;System made read sensor request and sensor failed.(task (name activate-zone) (object ?zone))

?e <- (event (text \"Read sensor\") (parameter ?zone) (status matched))

?nsrwm <- (num-sensor-reads (value ?nsr&:(< ?nsr ?*max-sensor-reads*)))(zone (name ?zone))

?er <- (event-response (event-type \"Read sensor\") (parameter ?zone)

(outcome fail) (status wait))

=>

(modify ?nsrwm (value (+ ?nsr 1))) (bind ?*sensor-read-result* -1)(modify ?e (status processed))(modify ?er (status done)))

Figure6:JESSrule

Animportantpartoftestdesignistotestthetestharnessitself.Wecandothisbyprovidingthetestharnesswithstreamsofexpectedandunexpectedevents(simulatingthesystem).Thisevaluationwillbothprovideusefulinfor-mationabouttherequirements,suchaswhethertheymakesense,aswellashelpmakethetestharnessrobustandreadyfortesting.

4.2TestGeneration

Anotherpreparatorystepistogeneratethetestcasesthatgivespecificdataforeachofthetestscenarios.Thetestscenarioscorrespondtoparticularpathsthroughtherequirementsscenario,andprovideinformationabouteventre-sponsesalongthosepaths.Thetestscenarioisbasicallyanabstractdescriptionofthetestcase.Forexample,intherequirementsscenarioinFigure3apar-ticularpathcouldbechosenifthemoisturelevelisbelowthecriticalmoisturelevel(seepreconditionforSequence2).Thetestscenariocorrespondingtothatparticularpathcontainsthatinformationwithoutdecidingconcretelywhatthemoisturelevelis.Atestcaseforthistestscenarioisafurtherspecializationandhasconcreteinputdata,suchasaparticularinstanceofasensorandamoisturelevelof10%whenthecriticalmoisturelevelis50%.Anevent-responseelement

10

inaparticulartestcaseforthistestscenariowouldthereforelooklikethis:(event-response(event-type“Readsensor”)(parameterS1)(outcome10)),whereevent-typeistheoutputeventfromthesystemtothetestharness,andoutcomeistheinputdatafromtheharnesstothesystem.ParticularobjectssuchassensorS1areinstantiatedinstancesoftheScenarioMLontologyinstanceTypes.Theorderofevent-responsesisderivedfromthesequenceofeventsinthetestscenario.Therecouldforexample,betwo“Failedtoreadsensor”eventsfollowedbyone“Successtoreadsensor”event;thedriverkeepstrackofappropriatere-sponsestoeachevent.Figure7showsatestcaseforthethescenariosnippetinfigure3thatusesfoursensorsandthereforecoversfourofthefivepathsofthescenario.Althoughtestgenerationwasmanualinourstudy,webelieveitispossibleingeneraltoautomatethegenerationofbothtestscenariosandtestcases.

(event-response (event-type \"Read sensor\") (parameter S1) (outcome fail))(event-response (event-type \"Read sensor\") (parameter S1) (outcome fail))(event-response (event-type \"Read sensor\") (parameter S1) (outcome 20))(event-response (event-type \"Read sensor\") (parameter S2) (outcome fail))(event-response (event-type \"Read sensor\") (parameter S2) (outcome fail))(event-response (event-type \"Read sensor\") (parameter S2) (outcome fail))(event-response (event-type \"Read sensor\") (parameter S3) (outcome 60))(event-response (event-type \"Read sensor\") (parameter S4) (outcome 40))(event-response (event-type \"Read sensor\") (parameter S4) (outcome fail))...

Figure7:Testcasesnippet

4.3TestExecution

Whenweexecuteatest,thetestharnessprovidesinputtothesystem,whenthesystemproducesoutput(resultsorfunctioncallstoexternalfunctions)thetestharnessmonitorstheseandresponds.Infigure8wedemonstratetestexecutionbyfollowinganexternalcallfromthesystemtoahardwaredevice(asensor)throughthetestharnessandbacktothesystem.Thetestharnessactsasasimulatedhardwaredeviceby“pickingup”thefunctioncallfromthesystemandregisteringthisasanevent(“Readsensor”).Thescenariorecognizercheckswiththetestcasewhetherornotthiseventwasexpected.Thedriverwillrespondtothecalloncethescenariorecognizerhasdecidedthattheeventwasmatched.Inourexample,thetestcasespecifiesabranchofthescenariothatcorrespondstoafailedsensorreadandthedriverthereforereturnsa-1,insteadofamoisturelevel,thatthesystemwillinterpretasafailedread.

Iftherearemismatchesbetweentheexpectedandreceivedevents,thesce-nariorecognizerwillfindoutandalertthetesteroftheparticularmismatch.

11

:Systemsensor.read();return -1:Test Harness APIevent: read sensor (s):ScenarioRecognizerread result: -1:Driverreceived eventreturn: =sensor failedexpected event:Test CaseFigure8:Testexecutionflow

Thistestingapproachallowsustoverifywhetherornotthescenariosarebeingfollowed.Iftheyarenotbeingfollowed,ithelpsusdeterminehowandwhy.

4.4TestEvaluation

Ourtestingapproachevaluateswhetherornotthesystempassestestcasesthatarederivedfromtherequirements.Wecandetecttwodistincttypesofmismatchesbetweentheactualsystembehaviorandtherequirements.First,thescenariorecognizerdetectsmismatchesbetweenindividualpairsofsystemeventsandeventsexpectedaccordingtotherequirementsscenarios.Second,weuseoraclesforevaluatingmismatchesthatcannotbededucedfromtheeventstreamalone.Anexampleofsucharequirementis:“AquaLush’swaterusagedoesnotexceedthezoneallocationforanyzone”.ThisrequirementshowsupasapreconditionforasequenceofeventsinIrrigateScenario.Wecannotverifythisbyasingleinstanceofanevent,butwecanverifythatthisconditionistrueeventuallybycomparingtheamountofwaterthatourtestcaseexpectseachzonetousewiththeactualamountofwaterused.

4.5Automation

FromScenarioMLwewill(ultimately)beabletogeneratebothtestcasesthatcoverallbranchesofascenario(reducingeachiterationtoaspecificnumberofrepetitions)andJESSrulesfordrivingandevaluatingthetests.Therulesnecessarytoperformanddrivethetestsareindependentofanyparticulartestcase,andthetestcaseisintheformofalistofevent-responseelementswhichwouldbeeasytogeneratefromamoresimplystructuredartifact,suchasatestscenario.WhatmakestheJESSrulesparticularlyinteresting(andgiveshopeforautomation)isthatthereisadirectrelationbetweentherulesandrequire-ments.Thatiswherethepowerliesintermsofdirectlyexposingrequirementsviolations.

12

5ValidationStudy

Inthefollowingsectionswewilldescribeourexperienceconstructingthetestharness,generatingtests,andrunningthetestsonasamplesoftwaresystem,AquaLush[6].

5.1AboutAquaLush

AquaLushisanautomaticirrigationsystemthatcontrolsirrigationbasedonsoilmoisturelevelsratherthantiming.Theprojectanditsartifactsareavailablein[6],whichprovidesmaterialsuchasrequirements,usecases,variousdesignmodels,andapproximately10KLOCofJavacode(includingthesimulator).WechoseAquaLushforseveralreasons;itisanadequatesizesystemwithafairamountofcomplexity;itwaspublishedwithafullrangeofrequirementsanddesignartifacts,anditwasdevelopedelsewhereforadifferentpurposethanours.

5.2Procedure

Weconstructedagoalgraphfromtheprojectmissionstatement,stakeholder-goalslist,andtheneedslist,andweusedtherequirementsandusecasestoproduceScenarioMLscenarios.Wecreatedsevendifferentscenariosbasedonsevenoftheeightexistingusecasesandrelatedthesescenariostogoalsinthegoalgraph.Weselectedalltheusecasesexceptfortheonethatdescribesthesimulationofthesystem.Weexcludedthisusecasebecauseitdoesnotdescribethefunctionalityofthesystemundertest,andinfactwewillusethesimulationaspartofourtestharness.Wewilluseoneofthesescenarios,IrrigateScenario,todescribeourwork.Wechosethisscenariobecauseitdescribesthemainfunc-tionalityofthesystemandrelatesittoseveralstakeholdergoals.Itisthereforeessentialthatthisrequirementisimplementedcorrectlyandtestedsufficiently.IrrigateScenariocontainstwomajoreventsequenceswherethesecondoneitselfconsistsofthreeeventsequences.Thefirsteventsequencedescribesreadingallthesensors.Thesystemwilltrytoreadeachsensoratmostthreetimes.Ifitcannotgetasuccessfulreadwithinthosethreetriesthesystemwillreportthatthesensorhasfailed.Thisistheeventsequencedescribedinfigure3.Inthesecondeventsequencethesystemwill(1)openallvalvesinazonethatneedsirrigation,(2)readthezone’ssensoronceeveryminuteuntilirrigationmuststop(eitherthezonereacheditscriticalmoisturelevel,usedupitszonealloca-tion,orthesensorfailedtogetareadwithinthreetries),and(3)closeallofthezone’svalves.Afterthat,thesystemwillopenthevalvesinanotherzonethatneedsirrigation,ifoneexists.Eachvalvecanbemanipulatedunsuccessfullyatmostthreetimes,afterwhichthevalveisreportedtohavefailed.

TomaketheexistingAquaLushsimulatorpartofthetestharness,wemodi-fieditscodetomonitoreventoccurrencesandprovidethesystemwithtestcaseinputratherthantheinputthattheoriginalsimulatorused.Wealsoadded

13

codetoconnectthesimulatortotherestofthetestharness.Wethencon-structedtestscenarios,testcases,driver,scenariorecognizer,andoracles,asdescribedintheprevioussection.

WeranourtestcasesagainstthereleasedversionofthesystemaswellasseveralmutantswithseededfaultscreatedbyMuJava[10].AcompatibilityproblembetweenMuJavaandJava1.5forcedustoconverttheAquaLushsys-temtoJava1.4.Wechangedeachoccurrenceofthenewenhancedforlooptotheregularforloop,eachoccurrenceofanenumeratedtypetoacollectionofageneraltype,andtypecaststoaccesstheseobjects.Theseweretheonlychangesinthesourcecodeofthesystemundertest.Theresultsfromthedifferenttest-ingactivities(testdesign,testgeneration,testexecution,andtestevaluation)arereportedbelow.

5.3TestDesign

OurtestingapproachreliesonrequirementsintheformofScenarioMLscenarios.Ingeneral,wedothetranslationfromusecasestoScenarioMLscenariosintwosteps.Inthefirststep,theusecasesaredirectlytranslatedintoScenarioML.Wewillrefertotheseastheoriginalscenarios.Inthesecondstep,theScenarioMLscenariosarerefinedbyimprovingtherequirements.Wewillrefertotheseastherefinedscenarios.Thesecondstepwaspartofthetestdesign.

Whenattemptingtomapscenarioeventsfromtheoriginalscenariostoout-putfromthesystem,wefoundthatseveraloftheeventsinthescenariowerenottestableintheiroriginalform.Onesucheventis“AquaLushplacesthezoneonanactivezonelist”,withtheconditionthatthesensorofthezonewasabletosuccessfullyreadthemoisturelevelandthatthemoisturelevelwasbelowthecriticalmoisturelevel(meaningthatthezoneneedswatering).Thecondi-tionplusthescenarioeventrepresentoneoftheoriginalrequirements.Atfirstglancethisrequirementseemedappropriate.However,asweweretryingtomapthescenarioeventtoaneventmeasurablefromtheboundaryofthesystem,werealizedthatthisrequirementisactuallymoreofadesignchoicethanarequire-ment.Aftersomediscussion,weconcludedthatthedeeperrequirementbehindthisdesignchoiceisthatonlyzoneswithworkingsensorsandmoisturelevelsbelowthecriticallevelshouldbeirrigated.Wethereforerefinedthescenariobydeletingthenon-testableinternaleventandreplacingitwithacheckofthesensorandmoisturelevelconditionsaspreconditionsoftheirrigationsequenceofthescenario.Thisrefinementofthescenariorepresentsthetruerequirementthatstateswhatthesystemshoulddo,ratherthanhowitshoulddoit.Thuswemanagedtoclarifytheoriginalstatedrequirementand,atthesametime,identifyarequirementthatistestablewithourblack-boxapproach.

SincewehadaccesstoAquaLushanditssimulator,weknewthataddinganitemtotheactivezonelistwasnotatestableeventfromtheboundaryofthesystem.However,whenanalyzingtherequirementfromarequirementsperspective,itwasclearthatitshouldnotbearequirementbecauseitis,infact,adesignchoice.Inthecaseswherethescenarioeventsandconditionsrepresenttruerequirements,wedefinethetestharnessAPItospecifywhatshouldbe

14

testablefromtheboundaryofthesystem.Thedesignersofthesystemshouldthuslookattherequirementstodesignthesystemtobetestable.

Anothernon-testableeventinoneoftheoriginalscenariosis“AquaLushcountsthenumberofworkingvalvesinallactivezonesandaddsthemtothetotalnumberofworkingvalves”.Thisisalsoadescriptionofhowtodosomethingratherthatwhatitshoulddo.Therequirementbehindthisdesignchoiceis,webelieve,thatAquaLushdoesnotuseupmorewaterthanitszoneallocationforanyzone,andthatthezoneallocationiscomputedbythevalveallocationmultipliedbythenumberofworkingvalvesinthezonetobeirrigated.Thevalveallocationiscomputedbythewaterallocationfortheirrigationcycledividedbythenumberofworkingvalvesintheentiresystem.Thisformoftherequirementstateswhatthesystemshoulddo,nothow,andprovidesussomethingtotest.Therecouldbeseveraldifferentwaysofdesigningthesystemtomeetthisrequirement.Onedesignchoiceistousethenon-testableeventwedescribedaboveandcountthenumberofworkingvalvesinallactivezonesandaddthemtothetotalnumberofworkingvalves.Anotherdesignchoicecouldbetokeepacounterofalltheworkingvalvesandupdatethiscountereverytimeavalvefailsorgetsrepaired.Weconcludedthattherequirementinitsoriginalformwasaprematuredesigndecisionandreplacedtheeventintherefinedversionofthescenariowithconditionsontheirrigationsequenceaswellasontologydefinitionsofwaterallocation,zoneallocation,andvalveallocation.Thereisanotherpeculiarityrelatingtotheoriginalscenarioandtheevent“AquaLushcountsthenumberofworkingvalvesinallactivezonesandaddsthemtothetotalnumberofworkingvalves”.Theeventwasplacedbeforetheeventofopeningthevalvesinthezonetoirrigate.Sincethezoneallocationisdependentonthenumberofworkingvalvesandthevalveshavenotyetbeenopened,thezoneallocationcomputedbythesystemcouldturnouttobebasedonfaultyassumptions.Inourrefinedscenario,sincethecalculationofthezoneallocationisnolongeraneventbutacondition,itisperformedintermsofthecurrentstateoftheworld.Aswewilldescribeinsection5.6,ouroracleswillcatchmismatchesbetweenthezoneallocationusedbythesystemandtheexpectedzoneallocationaccordingtoourinterpretationoftherequirements.Thereareseveralotheroriginalscenarioeventsthatturnedouttobenon-testable.Ineachcaseweevaluatedwhetherornottheassociatedrequirementreallymadesenseasarequirement.Ineveryinstancewherewejudgedtherequirementtobebadlyformed,wewereabletorewritetherequirementsothatpreviouslyassociatednon-testablescenarioeventscouldbeeliminated.Theresultwasabetterrequirement(describingawhatinsteadofahow)thatwouldbetestablewithablack-boxapproach.

Checkingrequirementsfortestabilityisonewayofvalidatingtherequire-mentsthroughtestdesign,butcreatingtestsfromtherequirementscouldalsorevealotherpotentialproblemswiththerequirements.Thesequenceofeventsintheoriginalscenariofailedtoaddresswhathappensifallvalvesinazonefailtoopen.Weinterpretedtherequirementtobethatthesystemshouldnotattempttoirrigateazoneinwhichnovalvesopened.However,whenweransuchatestcaseagainstourrequirementswefoundamismatch.Itturnsout

15

thatthesystemdoesattempttoreadthesensorinthezoneatleastoncebeforemovingontoopenthevalvesinthenextzonetobeirrigated.Thismismatchiseitheranimplementationfaultorarequirementsproblem.Inanycase,therequirementasstatedbytheoriginalscenariodidnotofferanunambiguousinterpretation.Therewerealsoseveralotherconceptsthatwerenotwellex-plainedintheoriginalscenarios.Itis,forexample,unclearwhatanopenvalveinAquaLushmeans.Ifopenmeansthatwaterisflowingoutofthevalve,whathappensifthevalvefailstoclose?TheoriginalusecasesaysthatthefailureisreportedtoPersistentStoreandthattheusecasethencontinues.Butifthewaterisflowing,wouldn’tthezoneeventuallyflood?Ifitmeansthatwatermightormightnotbeflowingwhenthevalveisopen,howcomethereisnoinformationabouthowthesystemcontrolsthewaterflow?Thereareothersim-ilarambiguities,thatarenotnecessarilyproblems,butwhichhavethepotentialtobeproblems.Hadwebeentestersusingourrequirements-basedtestingap-proachinthisproject,duringtheprojectdevelopment,wewouldhavehadachancetoclarifytheserequirementsbeforedesigners,developers,andtestersmaketheirownassumptionsofwhattheymean.

5.4TestGeneration

FromIrrigateScenariowederived6testscenarios.Wesettheinitialparametersoftheteststousefourzones.Eachzonehasexactlyonesensor(givenintherequirementsforAquaLush),andweinstantiatedeachzonetohavefourvalves(azonecanhavefrom1to32valves).Wesetthecriticalmoisturelevelto50%andthemaximumwaterallocationto10000gallons.Withthesesettings,the6testscenariosweresufficienttocoverall20branchesoftherequirementsscenario.Inthecaseofbranchpre-conditionsexpressingalternativeconditions,ourtestscenarioscovereachalternativecondition.Forexample,inthecaseofclosingthevalvesafterirrigation,thesystemcanchoosethebrancheitherbyreachingcriticalmoisturelevel,byusingupthezoneallocation,orbyfailingtoreadthesensorthreetimes.

Wecreated6testcasesoutofthetestscenarios,eachinstantiatedwithdifferentconcreteinputdata,forthesystemundertesttofollow(seesampletestcaseinfigure7).

Evenbeforerunningthetestcases,inthissituationweknowthatourtestscenarioscover100%ofthepathsintherequirementsscenario.WealsoknowthatweareexercisingseveralpartsoftheScenarioMLontology.Oncewehaverunthetestcaseswewillalsobeabletoreportonsuccessesorfailuresofparticularpaths,andprovideanestimateofhowwellthesystemmeetsitsrequirements.

5.5TestExecution

Duringtestexecution,weinterceptsystemoutputandusethistoinferoccur-rencesofevents.Onceaneventhasoccurreditismatchedagainsttheexpectedeventinthetestcase.Iftheeventismatched,thetestharnesswillmarkoff

16

thiseventfromthetestcase,provideinputdatatothesystemifnecessary,andthenmarkthenexteventtobeexpected.Ifthesystemeventdoesnotmatchtheexpectedeventinthetestharness,thetestharnesswillalertusofthemismatch.

Weranthesixtestcasesonthereleasedsystemandrecordedthedata.WethenranthesixtestcasesoneightdifferentmutantsthatwegeneratedbyusingMuJava[10].Althoughwegeneratedandtriedmanymoremutants,wechosetoreportonlytheseeightmutantsbecauseofthetypesoferrorstheycontain.Wedonotreportresultsfromanymutantsthatcausedthesystemtocrash.ExamplesofsuchmutantsareonesthatmanipulateloopcountersthatcausenullPointerExceptions,suchas

for(inti=0;++iinsteadof

for(inti=0;iWealsodecidedtoexcludethemanymutantsinwhichthemutationaffectedcodethatourtestscenariosdonot,andshouldnot,exerciseandcouldthereforenotdetect.Anothertypeofmutantwedecidedtoexcludefromthereportisonethatmanipulateswaterallocationandwaterusage,becausewehadalreadyfoundafaultinthecalculatedzoneallocationinthereleasedsystem.Mutantsofthatsortwould,therefore,notprovideanynewknowledgeofthetypesoffaultsthatourtestingapproachcanfind.Furthermore,mostofthosemutantswereacceptableaccordingtoourrequirementsanyway.Sincetheoriginalscenarioforirrigationdoesnotspecifyclearlywhenirrigationstopswithregardtoasensorread,weallowoneminutemoreorlessfromwhatouroracleexpects.Themutantsthatmanipulatedwaterusageandwaterallocationmostlyalteredthesevaluesbyone(+1,-1gallon),whichmadethemutantsuninterestinginanycase.

Theeightmutantswechoseareshowninfigure9.

Mutation name12345678AutoCycle.start()-AOIS_7AutoCycle.assignZoneAllocations()-AOIS_59Sensor.isFailed()-COI_3Sensor.setIsFailed()-COI_9Zone.isIrrigated()-ROR_14Zone.closeAllValves()-COD_8Zone.openAllValves()-COD_7Zone.setCriticalMoistureLevel()-AOIS_42Original codefor (int k = 0; k < zones.size(); k++)Mutated codefor (int k = 0; k++ < zones.size(); k++)Consequenceskips a zone when beginning irrigationskips to assign zone allocation to some zoneswitches failed and working sensorsswitches failed and working sensorssets zone to be irrigated prematurelyattempts to close non-working valvesattempts to open non-working valvesreduces the critical level by onefor (int m = 0; m < zones.size(); m++)for (int m = 0; m++ < zones.size(); m++)return isFailed;isFailed = newValue;if (criticalLevel <= sensor.read())if (!v.isFailed())if (!v.isFailed())criticalLevel = theLevel;return !isFailed;isFailed = !newValue;if (criticalLevel > sensor.read())if (v.isFailed())if (v.isFailed())criticalLevel = --theLevel;Figure9:Mutations

17

5.6Testevaluation

Thetestharnesshasseveralcomponentsforevaluatingthetestresults.Thescenariorecognizerwillrevealmismatchesbetweenexpectedandactualbehaviorintheformofmonitoringsystemeventsandcomparingthesetotestscenarioevents.Theoraclesexisttocheckthingsthatindividualsystemeventsdonotreveal.

Itisalsoworthnotingthatsomemismatchescanoccurbecausetherequire-mentsarewrong,orbecauseoffaultsinthetestharness.Wehopethatthesetypesoffaultswouldbeminimalwhenusingthisapproach,becausetheap-proachvalidatestherequirementsearlyandbecausethetestharnesscanbeconstructedwithoutaccesstothesystemundertest.Wecansimulatetheuseofthetestharnessbygivingiteventsequencesandhopefullyflushoutmosttestharnessfaultsbeforetestingofthesystem.5.6.1

TestingReleasedVersion

Wedetectedasystemeventmismatchwhenweranourfirsttestcaseonthereleasedversionofthesystem(seetestoutputbelow).Forthistestcase,thetestharnessmadethesystemthinkthatsensorS1failedtoreadthreetimesinthefirsteventsequenceofthescenario.ThesystemreadtheothersensorsandthenmovedontoopenthevalvesinthezonewithsensorS3.Oncethevalveswereopen,thesystemshouldhavestartedreadingsensorS3everyminuteuntilirrigationstopped.WhathappenedinsteadwasthatafterthevalvesopenedinthezonewithsensorS3,thesystemattemptedtoreadsensorS1,eventhoughthissensorwasnotinthezonethatwasbeingirrigatingatthetimeandhadactuallybeenreportedasnotworking.ThesecondtestcaseweranshowedthesamemismatchandithadalsobeendeterminedthatsensorS1wasnotworkingduringthefirsteventsequenceofthetestcase.Thismismatchshowedthatthesystemdidsomethingitshouldnothavedone....

AnopenvalveeventforV10wasreceivedasexpected.

Anunexpectedrequesttoreadthesensor

fromzoneS1wasreceived.Itwasunexpected

sincethesystemiscurrentlyirrigatingzoneS3.AreadsensoreventforzoneS3wasreceivedasexpectedwhileirrigating....

Wealsoneedtodetectotherformsofmismatcheswithouroracles.IntheScenarioMLontology,wedefinedthewaterallocationtobethemaximum

18

amountofwaterthatcanbeusedinanirrigationcycle(overallzones).Wedefinedthevalveallocationtobethewaterallocationdividedbythenumberofworkingvalvesandthezoneallocationtobethevalveallocationmultipliedbythenumberofworkingvalvesinaparticularzone.Weknowthatweneedtocalculatethezoneallocationbeforeazonebeginstogetirrigated,becauseweusereachingthezoneallocationasoneofthepossibleconditionsforstoppingtheirrigationofazone.Wethereforeimplementedanoraclethatcomputesthesevaluesbasedonthenumberofworkingvalvesandthenumberofzonesneedingtobeirrigatedatanygiventimeduringthetest.Wealsoknowthatthewaterusageinazoneistheamountoftimethatthezonehasbeenirrigatingmultipliedbytheflowrateinthezone.Usingthisinformationouroraclecouldinferwhetherornotthesystemhadcalculatedthezoneallocationcorrectly.Fromlookingattheoriginalscenario,wealreadyhaveseveralindicationsthatthiscalculationmightbefaulty,sinceitwasexpressedasaneventbeforethesequenceofopeningvalvesinazone.Also,thepreviousmismatchoftryingtoreadasensorinazonewhosesensorwasreportedtobeoutoforder,couldpotentiallyleadtoanincorrectcalculationofthezoneallocationifthatzone’svalveswereusedinthecalculationforthevalveallocation.Whenweranthethirdtestcase,atestcasethatusesuptheentirezoneallocationbecausethemoisturelevelneverreachesthecriticalmoisturelevelintwoofthefourzones,theoracledetectedamismatchintheamountofwaterusedandtheamountofwatertheoracleexpectedthesystemtouse(seetestoutputbelow).Thewaterallocationissetto10000gallons,andtherearefourzones.Thesystemstartedtoirrigatethesecondzone,inwhichoneofthevalvesfailedtoopen.Theoraclecalculatesthevalveallocationtobe10000gallonsdividedby15workingvalves,thencalculatesthezoneallocationtobethevalveallocationmultipliedby3,whichisthenumberofworkingvalvesinthezone.Thezoneallocationforthefirstzonetobeirrigatedistherefore2000gallons.Thesystemcontinuestoreadthesensoruntilithasusedup3312gallonsofwater,whichismorethanouroracleexpected.Laterintheeventtrace,wedetectedthatthesystemuseduplessthantheoraclewasexpecting(seesecondtestoutputbelow).Anothertestcaseshowedthatourinterpretationofthezoneallocationrequirementmakessense.Inthattestcase,onezoneisnotirrigatedbecauseitssensorfailstoreadandtwootherzonesarenotirrigatedbecausetheirmoisturelevelsareabovethecriticalmoisturelevel.Ouroraclecorrectlycalculatesthatthelastzoneshouldhavetheentirewaterallocation(10000gallons).Theotherthreetestcasesalsoshowedmismatchesofattemptingtoreadafailedsensorduringirrigationofanotherzoneandnotcalculatingthezoneallocationcorrectly....

AreadsensoreventforzoneS2wasreceivedasexpectedwhileirrigating.

StartclosingvalvesinzoneS2becausezoneusedupitszoneallocation.

19

zoneused:3312gallonsofwater,zoneallocation:2000.0.

AclosevalveeventforV06wasreceivedasexpected.......

AreadsensoreventforzoneS4wasreceivedasexpectedwhileirrigating.

StartclosingvalvesinzoneS4becausezoneusedupitszoneallocation.

zoneused:2256gallonsofwater,zoneallocation:3320.0.

AclosevalveeventforV15wasreceivedasexpected....5.6.2

TestingMutants

Wefoundmismatchesinalleightmutantsthatwearereportingoninthispaper.Asdescribedpreviously,theseededfaultsthatourapproachdidnotfindwereeitherfaultsthatcrashedthesystem,validimplementationsaccordingtotherequirements,alreadyreportedfaultsinthesystem,ornotinthecodethatthetestscenariosexercise.

Thefirstmutantskipssomezoneswhenbeginningtheirrigationscenario.Themutationhasalteredthecounterintheforloopfromfor(intk=0;k20

AreadsensoreventforS3wasreceivedasexpected.

AreadsensoreventforS3wasreceivedasexpected.

AreadsensoreventforS3wasreceivedasexpected.

AnunexpectedOpenvalveeventwithparameterV16wasdetected.

Thiswasunexpectedbecausethesystemis

currentlyattemptingtoactivateazonewhichmeansitisexpectingareadsensorevent.

Thesecondmutantusesasimilarloopfaulttoskipsomezoneswhenas-signingzoneallocations.Ourscenariorecognizerfoundthismismatchascanbeseeninthetestoutputbelow.Thesystemreadthesensoranddeterminedthatazoneneededirrigation.Itthensuccessfullyopenedthevalvesinthatzone.Thescenariorecognizerwasexpectingtoreceivea“Readsensor”eventnextaspartoftheirrigationsequence,butthesystemoutputa“Closevalve”eventinstead.Apparently,thesystemcouldnotcheckthezoneallocationforthiszoneanddecidedtoclosethevalvesandmoveon....

AnopenvalveeventforV09wasreceivedasexpected.

AReadsensoreventwasexpectedbutaClosevalveeventoccurred.

Thethirdmutantswitchesworkingandnon-workingsensors.Ourscenariorecognizerfoundthismismatchrightaftertheinitialsequenceofreadingthesensors.ThesystemoutputeventstoreadthesensorsandS1failedtoreadthreetimes.Themutatedsystemhoweverthinksthatthissensorisworkingandthereforeoutputsan“Openvalve”eventforthatzone.Thescenariorecognizerdetectsthismismatch,becausethetestcasehasalreadydeterminedthatsensorS1doesnotwork.Areadsensorexpected.

Areadsensorexpected.

Areadsensorexpected.

Areadsensorexpected.

Areadsensor

eventforS2wasreceivedaseventforS1wasreceivedaseventforS1wasreceivedaseventforS1wasreceivedaseventforS4wasreceivedas

21

expected.

AreadsensoreventforS3wasreceivedasexpected.

AnunexpectedOpenvalveeventwithparameterV02wasdetected.

Thiswasunexpectedbecausethesystemhasconcludedthatthezoneisinactive.

Wefoundsimilartypesofmismatchesfortherestoftheeightmutantsaswell.

5.7Results

Wehavedemonstratedtheeffectivenessofourapproachtorequirements-basedtestingbyidentifyinganumberofwaysinwhichthepublishedversionoftheAqualushsystemdoesnotsatisfyitsrequirements.Thetracesproducedbythetestharnesscomparingtheactualsystembehaviorwithexpectedbehaviorclearlyindicatethenatureoftherequirementsviolationsineachofoursixtestcases.ThismeansthattherequirementsscenarioIrrigateScenariowasnotachievedbythesystemandthatthehigh-levelstakeholdergoalsthatdependonthisscenariowerenotsatisfied.

Wehavefurthervalidatedourapproachbydemonstratingthatwecandetecterrorsthatresultfrommutationsofthesystem.Ourtestcaseswereabletodetectalltheseedederrors.

Finallywehavedemonstratedthatthereisgreatpotentialbenefitintheprocessofcreatingrequirements-basedtestsearlyoninthedevelopmentcycle.Wehaveshownthisbypresentingseveralexampleswherewhatappearedtobevalidrequirementsatthestartrevealedthemselvestobe,inreality,designchoices.ThesedisguiseddesignchoicesweredetectedwhiletryingtomaptheScenarioMLdescriptionsoftherequirementsontotheobservablebehaviorofthesystem.

Thestrengthofourapproachisderivedfromthesynergyoftwocomplemen-tarytechnologies.Ontheonehand,wehaveScenarioML.Itprovidesasemi-formallanguagefordescribingrequirementsthatisparticularlywellsuitedtothemechanizationoftheirmanipulation.Ontheotherhandwehavepattern-directedrule-basedprogrammingtechnologyasembodiedbyJess.WhatmakestheJESSrulesparticularlyinterestingisthatthereisadirectrelationbetweentherulesandrequirements.Thatiswherethepowerliesintermsofdirectlyexposingrequirementsviolations.Thedatastructuresusedtodriveourrule-basedprogramsaredirectlyderivedfromtheScenarioMLrepresentationsoftherequirements.Atthisearlystageoftheresearch,wederivethembyhand.ButourexperiencesofarhasconvincedusthattheprocessofderivingJESSrulesfromScenarioMLscenarioscanbecompletelyautomated.

22

6Conclusion

Inthisworkwehaveproposedrequirements-basedtestingasaparticularlyeffectivespecification-basedtestingapproach.Inourrequirements-basedtestingapproach,testsaredirectlydrivenbyandcheckedagainstrequirementsintheformofscenarios.Requirements-basedtestingutilizesthenaturalsymbioticrelationthatexistsbetweenrequirementsandtestingtoproducesoftwarethatbettersatisfiesstakeholderrequirements.

Ourevaluationstudyhasprovideduswithevidencethatourapproachiseffectiveinrevealingbothpotentialspecificationproblemsandimplementationfaults.Asstatedintheintroduction,atestingapproachalsoneedstobeeffi-cient.Inordertomakeourapproachbotheffectiveandefficient,wearecurrentlyworkingondifferentwaystoautomatetheprocess.

OnesubstantialtaskinourprocesshasbeentomanuallywritetheScenari-oMLscenarios.OurgroupiscurrentlyimplementinganEclipsepluginthatwilleasethistaskbyprovidingagraphicalinterfaceforeditingscenarios.WearealsoinvestigatingwaystoautomaticallygeneratetestscenariosthatcoverallbranchesofaScenarioMLscenario.WewilluseinstantiationsofinstanceTypeelementsinthescenarioontologyandpre-andpostconditionsasconstraintsandsearchthroughthescenariopathswiththoseconstraintsuntilwehaveenoughtestscenariostocoverallbranches.Asexplainedpreviouslythenumberofpar-ticularinstancesofaninstancetypecanimpactthenumberoftestscenariosneededtocoverallbranches.Wewillauto-generatetestcasesintheformofJESSevent-responsesfromthetestscenariosbyeitherrandomlyselectinginputdatathatsatisfiesthepath,orbymanuallyselectingdata.Wearealsolookingintowaysofautomatingtheproductionoftestdrivingcode,eventrecognizingcode,andoraclesfromScenarioMLscenarios.WeareawarethatwemightneedtoextendtheScenarioMLlanguagetobeabletoexpressinformationneededfortestingbutwhichwouldnotbenecessaryfortherequirementsthemselves.Wearealsoworkingonimprovementsforexecutingtestsandcollectionsoftestsautomatically.

Onceautomationisinplace,wewillperformamoresubstantialvalidationstudytoevaluatetheeffectivenessoftestcasesthatareauto-generatedfromScenarioML,Wealsowanttoevaluatetheefficiencyofusingourapproachwithautomationinplacecomparedtoothertestingapproaches.Inorderforsuchanevaluationtobeconvincingweneedtodevelopaframeworkforevaluatingdifferenttestingapproacheswithregardtoeffectivenessandefficiency.Itisgenerallyknownthatfaultsthatstemfromrequirementsmisunderstandingsareexpensivetofixlateinthedevelopmentcycle.Timespentonspecifyingandvalidatingtherequirementsforthesystemcouldthereforebegainedbyalowernumberofseverefaultslaterinthecycle.Weintendtodevelopaframeworkthatclassifiesthedifferenttypesoffaultsanapproachcanfindandmeasurestimespentonspecificationdevelopmentandtestingallactivities.

23

References

[1]J.F.Allen.Maintainingknowledgeabouttemporalintervals.CACM,

26(11):832–843,1983.[2]T.A.Alspaugh,S.E.Sim,K.Winbladh,M.Diallo,H.Ziv,andD.J.

Richardson.Theimportanceofclarityinusablerequirementsspecificationformats.TechnicalReportUCI-ISR-06-14,Inst.forSoftw.Res.,Univ.ofCal.,Irvine,2006.[3]T.A.Alspaugh,B.Tomlinson,andE.Baumer.Usingsocialagentsto

visualizesoftwarescenarios.InSoftVis’06,pages87–94,2006.[4]L.C.BriandandY.Labiche.AUML-basedapproachtosystemtesting.

SoftwareandSystemModeling,1(1):10–42,2002.[5]L.Chung,B.A.Nixon,E.Yu,andJ.Mylopoulos.Non-FunctionalRe-quirementsinSoftwareEngineering.Springer,2000.[6]C.Fox.IntroductiontoSoftwareEngineeringDesign:Processes,Principles

andPatternswithUML2.AddisonWesley,2006.[7]D.Graham.Requirementsandtesting:Sevenmissing-linkmyths.IEEE

Software,19(5):15–17,2002.[8]JESS(JavaExpertSystemShell).

jess/.

http://herzberg.ca.sandia.gov/

[9]A.v.LamsweerdeandL.Willemet.Inferringdeclarativerequirements

specificationsfromoperationalscenarios.IEEETrans.onSoftw.Eng,24(12):10–1114,1998.[10]Y.-S.Ma,J.Offutt,andY.R.Kwon.Mujava:anautomatedclassmutation

system.Softw.Test,Verif.Reliab,15(2):97–133,2005.[11]H.Muccini,M.S.Dias,andD.J.Richardson.Systematictestingofsoft-warearchitecturesintheC2style.InFASE’04,volume2984ofLectureNotesinComputerScience,pages295–309.Springer,2004.[12]C.Nebut,F.Fleurey,Y.LeTraon,andJ.-M.J´ez´equel.Automatictestgen-eration:Ausecasedrivenapproach.IEEETrans.Softw.Eng.,32(3):140–

155,2006.[13]T.O.O’Malley,D.J.Richardson,andL.K.Dillon.Efficientspecification-basedoraclesforcriticalsystems.InSecondCaliforniaSoftwareSymposium(CSS’96),Apr.1996.[14]C.Rolland,C.Souveyet,andC.BenAchour.Guidinggoalmodelingusing

scenarios.IEEETrans.onSoftw.Engineering.,24(12):1055–1071,1998.

24

[15]A.S.Tanenbaum.Indefenseofprogramtestingorcorrectnessproofs

consideredharmful.SIGPLANNot.,11(5):–68,1976.[16]E.J.Weyuker.Ontestingnon-testableprograms.TheComputerJournal,

25(4):465–470,1982.[17]M.W.Whalen,A.Rajan,M.P.Heimdahl,andS.P.Miller.Coverage

metricsforrequirements-basedtesting.InInternationalSymposiumonSoftwareTestingandAnalysis(ISSTA’06),2006.[18]K.Winbladh,T.A.Alspaugh,H.Ziv,andD.J.Richardson.Architecture-basedtestingusinggoalsandplans.InROSATEA’06,2006.[19]K.Winbladh,T.A.Alspaugh,H.Ziv,andD.J.Richardson.Anautomated

approachforgoal-driven,specification-basedtesting.In21stInternationalConferenceonAutomatedSoftwareEngineering(ASE2006),pages2–292,2006.[20]Q.XieandA.Memon.Designingandcomparingautomatedtestoraclesfor

GUI-basedsoftwareapplications.ACMTrans.onSoftw.Eng.andMethod.,2006.

25