Combinatorial GUI Testing:Automatic Test Case Generation for ...

Combinatorial GUI Testing:Automatic Test Case Generation for ...

Combinatorial GUI CS2310 Fall 2017 Seminar Testing: Automatic Test Case Generation for Android Apps Yuhuan Jiang [email protected] Papers Selected for This Talk MAIN PAPER 1) Reducing Combinatorics in GUI Testing of Android Applications. Mirzaei, Nariman and Garcia, Joshua and Bagheri, Hamid and Sadeghi, Alireza and Malek, Sam. ICSE 2016. https://dl.acm.org/citation.cfm?id=2884853 SUPPORTING PAPERS 2) Automated Test Input Generation for Android: Are We There Yet? Choudhary, Gorla, Orso. https://dl.acm.org/citation.cfm?id=2916273

3) Dynodroid: An Input Generation System for Android Apps. Aravind Machiry, Rohan Tahiliani, Mayur Naik. GUI TESTING 02/06/2020 COMBINATORIAL CS2310 Seminar Contents Background Android GUI Teminology Supporting Papers Popular Randomized Testing Frameworks Main Paper TrimDroid: Dependency-based Test Case Generation Thoughts & Comments Drawbacks of TrimDroid Comparisons

02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Image Source Android GUI Terminology Intent Data from First Activity can be passed into Second Activity in the payload of the intent Activity Input Widget Widget Event

02/06/2020 onClick() onLongClick() onFocusChanged() onKey() onTouch() onKeyDown() onKeyUp() ... COMBINATORIAL GUI TESTING CS2310 Seminar Supporting Papers The major paper compared their system with other testing frameworks

Two investicated: Monkey [1] Generates pseudo-random streams of events Dynodroid [2] Also generates events randomly, but is more capable and smarter about selecting events [1] https://developer.android.com/studio/test/monkey.html [2] https://dynodroid.github.io/ 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar

Monkey Comes with the Android developer toolkit. Sends random UI events to the app. Provides options to change random behavior: Restrict the testing to a specific package Restrict the type of events and how frequently they should be tested Adjust percentage of touch events, motion events, etc. Why provide these options? Customize the tests to suit differnet needs Reduce the number of test cases Random UI events Android Monkey 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Monkey: Limitations

Code coverage can be low Will see a comparison later in major paper Random selection strategy too basic Does not consider the context or the current state of the application Users have to specify max #events Once reached, Monkey will stop running This leaves many cases uncovered 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dynodroid Also a random generator of events UI events (clicks, drags, ) System events (Battey low, SMS received, ) Generates events with a observe-select-execute cycle:

Observer: Computes the set of events suitable for the current state of the app Selector: Selects from the set of events picked by the observer Executer: Executes the selected event 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dynodroid: Advantages & Limitations Advantages compared to Monkey: Considers both UI events + System events Only checks the relevant events for the app Monitors when the app registers a listener within the Android Framework Smarter random selection methods: Frequency: leasts frequently selected are first selected Biased: events relevant in more contexts are more likely selected

Limitations compared to Monkey: 5 Slower than Monkey The observer needs to query the view hierarchy Uses reflection, which is usually slow Needs a modified Android SDK Limited to a particular SDK version Migrating to other versions may require code changes Android is evolving fast! Dynodroid currently supports Android 2.3 Latest version as of this seminar is Android 8.1 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar A Running Example The Expense Reporting System (ERS) app: Contains 5 Activities NewReportActivity:

screen; main/entry ItemizedReportActivity: 02/06/2020 to report itemized meal expenses for a day; QuickReportActivity: to report aggregated meal expenses for a trip; ConfirmationActivity: to show the expense to the user for confirmation before submitting; SummaryActivity: To show a summary of the expense after COMBINATORIAL GUI TESTING CS2310 Seminar submitting. Explosion of Combinatorial States

10 STATES 2 STATES 1023 = 60 STATES 3 STATES 60384 = 23,040 STATES! 6 STATES 222 STATES 222 STATES 02/06/2020 1023 = 384 STATES COMBINATORIAL GUI TESTING CS2310 Seminar

TrimDroid Architecture 02/06/2020 COMBINATORIAL GUI TESTING 12 Input File: APK Not source code is required Source code is not necessary to statically analyze dependencies among widgets An APK file is used to install apps on Android Format: a ZIP file Contents related to GUI Testing: Example.apk/ AndroidManifest.xml Layout/

... MainActivity.xml ConfirmationActivity.xml AbcActivity.xml XyzActivity.xml

A typical manifest XML file

CS2310 Seminar Test Case Generation Example: Generating test cases for a prime path: # AT hDep(a2, e3) a0 Test cases for event e3 of activity a2 e1 Test cases for event e1 of activity a0 e3 Activity dependencies a2

Example prime path Test cases for the exam prime path 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation Three research questions: Q1 (Test Case Reduction): Can TrimDroid reduce the number of test cases? Q2 (Code Coverage): Can TrimDroid provide good code coverage? Q3 (Cross Comparison): How does TrimDroid compare to other techniques? Programs tested are from the F-Droid repo: [1]

Must be native apps (not apps which has a WebView) Apps must only use standard libraries and UI [1] https://f-droid.org widgets 14 apps are selected 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation: Test Case Reduction 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation: Code Coverage 02/06/2020

COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation: Cross Comparison 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar My Thoughts on TrimDroid Non-standard GUI Widgets Modern apps create their own custom widgets, which cannot be recognized by TrimDroid. Only discrete values are considered What about continuous inputs? E.g., a slider that goes continuously from 0.0 to 1.0; There are infinitely many values within the range. Only works on static GUI

UI elements generated programatically (dynamically) are not uncommon in Android app development. What About Screen Metrics? There is no way to test if a button or image is placed at the right position, has the right height, etc. What About Animations? Modern mobile applications often use animations. Need to test whether an animation is executed as expected. 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar My Comparison of Techniques TrimDroid Mechanism for reducing test cases

Monkey Dynodroid None Finds related events in each app state No No Need to compile with modified SDK Only the APK is needed. Nothing extra is needed. Comes

with the developer toolkit. Needs to use their modified version of Android SDK. Same as exhaustive Could miss cases due to randomness Better than Monkey due to smarter event selection strategy Dynamically Generated GUI?

Not Supported Supported Supported Event Types UI events only UI events + System events UI events + System events Source Code Needed? Ease of Use Coverage

02/06/2020 Latest Detects dependencies COMBINATORIAL GUI TESTING No claims. CS2310 Seminar Conclusion The key to automating GUI testing: Reduce the number of test cases, as exhaustive enumeration is intractable (natural of combinatorics). TrimDroid is a pricipled way of generating test cases: (in constrast with random approaches) Statically analyzes the APK for dependencies Groups dependent widgets into sets

Generates full combinations only for widgets in each set Desired properties of a GUI test case generator: Can reduce the number of test cases without sacrificing code coverage (TrimDroids achievement); Supports more types of events (UI + system) to maximize code coverage; Does not spend too long on generating test cases, otherwise the purpose of speeding up the actual testing is defeated; Does not need the source code; Can keep up with the evolution of the Android SDK. 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar References MAIN PAPER 1) Reducing Combinatorics in GUI Testing of Android Applications. Mirzaei, Nariman and Garcia, Joshua and Bagheri, Hamid and Sadeghi, Alireza and Malek, Sam. ICSE 2016.

https://dl.acm.org/citation.cfm?id=2884853 SUPPORTING PAPERS 2) Automated Test Input Generation for Android: Are We There Yet? Choudhary, Gorla, Orso. https://dl.acm.org/citation.cfm?id=2916273 3) Dynodroid: An Input Generation System for Android Apps. Aravind Machiry, Rohan Tahiliani, Mayur Naik. GUI TESTING 02/06/2020 COMBINATORIAL CS2310 Seminar

Recently Viewed Presentations

  • Physical Geography of Europe

    Physical Geography of Europe

    The Apennine or Italian Peninsula is a long, thin, boot-shaped piece of land on which the country of Italy lies. The Apennines mountain range, which includes the active volcano Mount Vesuvius, extends down the center of the peninsula. Only about...
  • Reconnaissance and Detection Observations

    Reconnaissance and Detection Observations

    Reconnaissance and Detection Observations ... C/B Equipment The Improved Chemical Agent Monitor was in inventory and used extensively for survey of suspected contamination M21 Remote Standoff Chemical Agent Alarm was available and deployed as part of FOX NBCRV sensor suite...
  • SCHOOL LEAVERS INFORMATION SESSIONS ARE YOU TAFE READY?

    SCHOOL LEAVERS INFORMATION SESSIONS ARE YOU TAFE READY?

    Independent Learner - not rely on others, time management skills. Understand your own disability - explain to people what you have, what are the issues and what support you would require. LL&N - entry requirements into course (all courses have...
  • R EAD N THINK Drake: Explorer or Pirate?

    R EAD N THINK Drake: Explorer or Pirate?

    Drake was also not your typical pirate. Though he had no problem stealing from the Spanish he treated the people he robbed rather well. In one case he invited the captain of a ship he was robbing to have dinner...
  • The many uses of enriched thesauri and ontologies

    The many uses of enriched thesauri and ontologies

    Support for conceptual question analysis Browsing a hierarchy to find search concepts Mapping from users' terms to descriptors or free-text terms Behind-the-scenes query term expansion: hierarchic and synonym expansion Especially important for free-text searching Provide a tool for indexing, esp....
  • Romeo & Juliet - Social Circle City School District

    Romeo & Juliet - Social Circle City School District

    Literary Terminology. Tragedy. A form of drama based on human suffering that invokes in the audience catharsis (the process of releasing strong or repressed emotions)or pleasure in the viewing (schadenfreude). So, why do we read tragedies? What to we take...
  • Embolism: - JUdoctors

    Embolism: - JUdoctors

    Embolism. An embolus is a detached intravascular solid, liquid, or gaseous mass that is carried by the blood to a site distant from its point of origin ... infarction) of downstream tissue. ... Pulmonary Hypertension and right ventricular failure ....
  • CALVIN: Optimization of Californias Water Supplies 29 November

    CALVIN: Optimization of Californias Water Supplies 29 November

    PowerPoint Presentation Water Demand Functions Calvin's Economic Requirements Data Base for the SWAP Model PowerPoint Presentation SWAP Model Regions PowerPoint Presentation Agricultural Crop Descriptions Agricultural Response to Changes in the Price and Quantity of Water PowerPoint Presentation Efficiency-Cost Trade-offs.