Combinatorial GUI Testing:Automatic Test Case Generation for ...
Combinatorial GUI CS2310 Fall 2017 Seminar Testing: Automatic Test Case Generation for Android Apps Yuhuan Jiang [email protected] Papers Selected for This Talk MAIN PAPER 1) Reducing Combinatorics in GUI Testing of Android Applications. Mirzaei, Nariman and Garcia, Joshua and Bagheri, Hamid and Sadeghi, Alireza and Malek, Sam. ICSE 2016. https://dl.acm.org/citation.cfm?id=2884853 SUPPORTING PAPERS 2) Automated Test Input Generation for Android: Are We There Yet? Choudhary, Gorla, Orso. https://dl.acm.org/citation.cfm?id=2916273
3) Dynodroid: An Input Generation System for Android Apps. Aravind Machiry, Rohan Tahiliani, Mayur Naik. GUI TESTING 02/06/2020 COMBINATORIAL CS2310 Seminar Contents Background Android GUI Teminology Supporting Papers Popular Randomized Testing Frameworks Main Paper TrimDroid: Dependency-based Test Case Generation Thoughts & Comments Drawbacks of TrimDroid Comparisons
02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Image Source Android GUI Terminology Intent Data from First Activity can be passed into Second Activity in the payload of the intent Activity Input Widget Widget Event
02/06/2020 onClick() onLongClick() onFocusChanged() onKey() onTouch() onKeyDown() onKeyUp() ... COMBINATORIAL GUI TESTING CS2310 Seminar Supporting Papers The major paper compared their system with other testing frameworks
Two investicated: Monkey  Generates pseudo-random streams of events Dynodroid  Also generates events randomly, but is more capable and smarter about selecting events  https://developer.android.com/studio/test/monkey.html  https://dynodroid.github.io/ 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar
Monkey Comes with the Android developer toolkit. Sends random UI events to the app. Provides options to change random behavior: Restrict the testing to a specific package Restrict the type of events and how frequently they should be tested Adjust percentage of touch events, motion events, etc. Why provide these options? Customize the tests to suit differnet needs Reduce the number of test cases Random UI events Android Monkey 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Monkey: Limitations
Code coverage can be low Will see a comparison later in major paper Random selection strategy too basic Does not consider the context or the current state of the application Users have to specify max #events Once reached, Monkey will stop running This leaves many cases uncovered 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dynodroid Also a random generator of events UI events (clicks, drags, ) System events (Battey low, SMS received, ) Generates events with a observe-select-execute cycle:
Observer: Computes the set of events suitable for the current state of the app Selector: Selects from the set of events picked by the observer Executer: Executes the selected event 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dynodroid: Advantages & Limitations Advantages compared to Monkey: Considers both UI events + System events Only checks the relevant events for the app Monitors when the app registers a listener within the Android Framework Smarter random selection methods: Frequency: leasts frequently selected are first selected Biased: events relevant in more contexts are more likely selected
Limitations compared to Monkey: 5 Slower than Monkey The observer needs to query the view hierarchy Uses reflection, which is usually slow Needs a modified Android SDK Limited to a particular SDK version Migrating to other versions may require code changes Android is evolving fast! Dynodroid currently supports Android 2.3 Latest version as of this seminar is Android 8.1 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar A Running Example The Expense Reporting System (ERS) app: Contains 5 Activities NewReportActivity:
screen; main/entry ItemizedReportActivity: 02/06/2020 to report itemized meal expenses for a day; QuickReportActivity: to report aggregated meal expenses for a trip; ConfirmationActivity: to show the expense to the user for confirmation before submitting; SummaryActivity: To show a summary of the expense after COMBINATORIAL GUI TESTING CS2310 Seminar submitting. Explosion of Combinatorial States
10 STATES 2 STATES 1023 = 60 STATES 3 STATES 60384 = 23,040 STATES! 6 STATES 222 STATES 222 STATES 02/06/2020 1023 = 384 STATES COMBINATORIAL GUI TESTING CS2310 Seminar
TrimDroid Architecture 02/06/2020 COMBINATORIAL GUI TESTING 12 Input File: APK Not source code is required Source code is not necessary to statically analyze dependencies among widgets An APK file is used to install apps on Android Format: a ZIP file Contents related to GUI Testing: Example.apk/ AndroidManifest.xml Layout/
A typical manifest XML file A typical layout XML file 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar
Interface Model An interface model contains information about all the GUI inputs of an Android app. Achieves this by extracting from XML files in APK 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Activity Transition Model An activity transition model (ATM) represents the high-level behavior of an apps GUI. Stores transitions resulting from invocations of the event handlers in the activities The key to test case generation. 02/06/2020
COMBINATORIAL GUI TESTING CS2310 Seminar Path Generation A simple path is a path with no cycle (the path itself is allowed to be a cycle). A prime path is a maximal simple path (i.e., cannot be extended without losing the simple property). 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dependency Extraction Type 1: Widget dependency Defined between: Two widgets: w1 and w2
Established when the combinations of the values of w1 and w2 affect an apps control-flow or data-flow. 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dependency Extraction Type 1: Widget dependency For our ERS app, we have: a0 a2 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar
Dependency Extraction Type 2: Handler dependency Defined between: a set of widgets w, and an event handler e Established when the values of w is is used in e This means: all combinations of the widgets in w and the event resulting in the invocation of e must be tested 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dependency Extraction Type 2: Handler dependency
For the ERS app a0 e1 a2 02/06/2020 e3 COMBINATORIAL GUI TESTING CS2310 Seminar Dependency Extraction Type 3: Activity Dependency Defined on: An activity a1 A set of widgets w in a1 Another activity a2 Established when the values of w in a1 may impact the
behavior of a2 This means: All combinations of the widgets in w must be tested 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Dependency Extraction Type 3: Activity Dependency For the ERS app a0 e1 a2 02/06/2020 e3 COMBINATORIAL GUI TESTING
CS2310 Seminar Test Case Generation Test cases for a prime path is generated by combing test cases of events in each activity Test cases for an event in an activity is generated by combining the values of each input class. 1. Generate for each event 2. Generate for each prime path 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Test Case Generation Example: Generating test cases for an event: Widget value
combinations for each dependent set Activity a0 of ERS Input classes for each input widget in a0 Handler dependency sets of e1 in a0 MERGE Simply make sure every unique value appears Test cases for event e1 of activity a0 02/06/2020 COMBINATORIAL GUI TESTING
CS2310 Seminar Test Case Generation Example: Generating test cases for a prime path: # AT hDep(a2, e3) a0 Test cases for event e3 of activity a2 e1 Test cases for event e1 of activity a0 e3 Activity dependencies a2
Example prime path Test cases for the exam prime path 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation Three research questions: Q1 (Test Case Reduction): Can TrimDroid reduce the number of test cases? Q2 (Code Coverage): Can TrimDroid provide good code coverage? Q3 (Cross Comparison): How does TrimDroid compare to other techniques? Programs tested are from the F-Droid repo: 
Must be native apps (not apps which has a WebView) Apps must only use standard libraries and UI  https://f-droid.org widgets 14 apps are selected 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation: Test Case Reduction 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation: Code Coverage 02/06/2020
COMBINATORIAL GUI TESTING CS2310 Seminar Evaluation: Cross Comparison 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar My Thoughts on TrimDroid Non-standard GUI Widgets Modern apps create their own custom widgets, which cannot be recognized by TrimDroid. Only discrete values are considered What about continuous inputs? E.g., a slider that goes continuously from 0.0 to 1.0; There are infinitely many values within the range. Only works on static GUI
UI elements generated programatically (dynamically) are not uncommon in Android app development. What About Screen Metrics? There is no way to test if a button or image is placed at the right position, has the right height, etc. What About Animations? Modern mobile applications often use animations. Need to test whether an animation is executed as expected. 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar My Comparison of Techniques TrimDroid Mechanism for reducing test cases
Monkey Dynodroid None Finds related events in each app state No No Need to compile with modified SDK Only the APK is needed. Nothing extra is needed. Comes
with the developer toolkit. Needs to use their modified version of Android SDK. Same as exhaustive Could miss cases due to randomness Better than Monkey due to smarter event selection strategy Dynamically Generated GUI?
Not Supported Supported Supported Event Types UI events only UI events + System events UI events + System events Source Code Needed? Ease of Use Coverage
02/06/2020 Latest Detects dependencies COMBINATORIAL GUI TESTING No claims. CS2310 Seminar Conclusion The key to automating GUI testing: Reduce the number of test cases, as exhaustive enumeration is intractable (natural of combinatorics). TrimDroid is a pricipled way of generating test cases: (in constrast with random approaches) Statically analyzes the APK for dependencies Groups dependent widgets into sets
Generates full combinations only for widgets in each set Desired properties of a GUI test case generator: Can reduce the number of test cases without sacrificing code coverage (TrimDroids achievement); Supports more types of events (UI + system) to maximize code coverage; Does not spend too long on generating test cases, otherwise the purpose of speeding up the actual testing is defeated; Does not need the source code; Can keep up with the evolution of the Android SDK. 02/06/2020 COMBINATORIAL GUI TESTING CS2310 Seminar References MAIN PAPER 1) Reducing Combinatorics in GUI Testing of Android Applications. Mirzaei, Nariman and Garcia, Joshua and Bagheri, Hamid and Sadeghi, Alireza and Malek, Sam. ICSE 2016.
https://dl.acm.org/citation.cfm?id=2884853 SUPPORTING PAPERS 2) Automated Test Input Generation for Android: Are We There Yet? Choudhary, Gorla, Orso. https://dl.acm.org/citation.cfm?id=2916273 3) Dynodroid: An Input Generation System for Android Apps. Aravind Machiry, Rohan Tahiliani, Mayur Naik. GUI TESTING 02/06/2020 COMBINATORIAL CS2310 Seminar
The Apennine or Italian Peninsula is a long, thin, boot-shaped piece of land on which the country of Italy lies. The Apennines mountain range, which includes the active volcano Mount Vesuvius, extends down the center of the peninsula. Only about...
Reconnaissance and Detection Observations ... C/B Equipment The Improved Chemical Agent Monitor was in inventory and used extensively for survey of suspected contamination M21 Remote Standoff Chemical Agent Alarm was available and deployed as part of FOX NBCRV sensor suite...
Independent Learner - not rely on others, time management skills. Understand your own disability - explain to people what you have, what are the issues and what support you would require. LL&N - entry requirements into course (all courses have...
Drake was also not your typical pirate. Though he had no problem stealing from the Spanish he treated the people he robbed rather well. In one case he invited the captain of a ship he was robbing to have dinner...
Support for conceptual question analysis Browsing a hierarchy to find search concepts Mapping from users' terms to descriptors or free-text terms Behind-the-scenes query term expansion: hierarchic and synonym expansion Especially important for free-text searching Provide a tool for indexing, esp....
Literary Terminology. Tragedy. A form of drama based on human suffering that invokes in the audience catharsis (the process of releasing strong or repressed emotions)or pleasure in the viewing (schadenfreude). So, why do we read tragedies? What to we take...
Embolism. An embolus is a detached intravascular solid, liquid, or gaseous mass that is carried by the blood to a site distant from its point of origin ... infarction) of downstream tissue. ... Pulmonary Hypertension and right ventricular failure ....
PowerPoint Presentation Water Demand Functions Calvin's Economic Requirements Data Base for the SWAP Model PowerPoint Presentation SWAP Model Regions PowerPoint Presentation Agricultural Crop Descriptions Agricultural Response to Changes in the Price and Quantity of Water PowerPoint Presentation Efficiency-Cost Trade-offs.
Ready to download the document? Go ahead and hit continue!