Getting Started

Welcome to Interaction Mining: a privacy-preserving, open-source project to collect organic mobile app task flows.

We enable collection of app task flows directly from your Android phone. We do this using an Android app through Accessibility Service API and our web application.

Our web application is written in Next.js, a React framework.

Check out our code below:

We plan on supporting iOS app task flow collection in the near future.

Want to Get Involved?

❗ This Section is Still Under Development

Create an Android App Task Flow

Requirements

  • Android phone with version 12.0 or higher
  • A Wi-Fi connection

Install Android App

Download the mobile app APK and install it on your Android device.

You may need to enable installation from unknown sources in your device settings. This setting allows you to install apps from sources other than the Google Play Store.

  1. Open the Settings app on your device
  2. Look for Security or Privacy settings (the name may vary depending on your device)
  3. Find the option called "Install unknown apps" or "Install from unknown sources"

Next, locate and open the APK file you downloaded. Follow the instructions from the installer to complete the installation.

Once the app is installed, a new app with an Android icon and the name ODIM will appear on your device's home page.

Create a New Task Flow

  1. Sign in to the web application and navigate to the user dashboard.
  2. Click on to redirect to new task creation form.
  3. Fill out the form with chosen app and description and click
  4. A QR code corresponding to the new capture will be generated.
  5. Open the Android app and scan this QR code to get task information.
  6. Follow on-screen prompts to enable accessibility permissions.

Note: Some phones may first ask you to grant app permissions. If this happens, go to Settings > Apps > ODIM and enable all necessary permissions.

Capture a Task From Mobile App

Launch the target app you want to interact with. The interaction mining app captures your screens and interactions in the background.

After ending a trace, you can:

  • View the list of screens of the task flow from the app interface
  • Select the recorded task flow and upload it from your device

Once uploaded, the screens should be detected by the web app. From there, you can complete the pre-processing workflow, which includes:

  • Tagging gestures and interactions
  • Redacting sensitive information from screenshots

Stopping ODIM Recording

To stop ODIM from recording entirely:

  1. Navigate back to Settings > Accessibility
  2. Toggle the ODIM app OFF

You can restart recording at any time by toggling ODIM back ON.

Post-Process on Web App

Once screens are uploaded, you must post-process the screens to fix errors and remove privacy sensitive information in the screens.

The post-processing interface uses a multi-part form with phases to:

  • Delete extraneous or irrelevant screens from a trace.
  • Tag missing or incorrect gestures and interactions.
  • Redact sensitive information from screens.

Delete Screens

To delete unneeded screens from your trace:

  1. Review your trace and locate extraneous screen(s).
  2. Click the icon on the top right of the screen.

Restore deleted screens by clicking on the top left

Tag Gestures

To complete the post-processing stage, all screens must be tagged with an gestures and a short 1-2 word description of the interaction.

Add a Gestures to Screen

Screens with missing gestures will overlay icon in the bottom filmstrip.

  1. Select the screen with missing gesture from the bottom filmstrip.
  2. On the top right view displaying the selected screen, click the area of screen where the interaction occurred.
  3. A colored circle will pop up with a dropdown of common interaction types, select the correct interaction type
  4. Write a short description of what the gesture does in the text area below.

Amend Incorrect Gestures

Some screens may have erroneous interactions and may need to be corrected.

  1. Select the screen with incorrect gesture from the bottom filmstrip.
  2. On the top right view displaying the selected screen, drag the colored gesture icon to where the interaction actually occurred on screen.
  3. Change the gesture type from dropdown and description from text area if needed.

Redact Screens

Private data can be redacted from screens by clicking and dragging over screen regions containing sensitive data.

Tools

The redaction editor provides three tools to help you redact personal data.

  • Move: Select (or click V) to move around and zoom in/out of the screen on the redaction canvas.
  • Draw: Select (or click P) to draw a redaction over the screen by click + drag.
  • Eraser: Select (or click E) to erase a redaction by clicking on a redaction box. You can also click the icon on the top left of the redaction.
  1. Select the screen with sensitive data from bottom filmstrip.
  2. On the top right view displaying the selected screen, click the draw tool and click + drag over bounding box containing sensitive data.
  3. Redact the bounding box area so the view hierarchy metadata of the element with personal data is also redacted.

View hierarchy metadata only redaction is currently in development

Annotate Task Flow

Review all the screen annotations to make sure everything is correct. Then, give a 1-2 sentence summary of the screens and click to submit the task flow.