Heuristic Analysis

Since there is no current way to analyze a comprehensive control system for diminished reality, as such a system does not currently exist, we evaluated the design of the HoloLens specifically looking at the methods for audio adjustment.

Three evaluators independently assessed the usability of each component listed against the heuristics and provided a rating. An additional rating was given in assessment of the component’s ability to control or display audio. These scores were based on a 0 to 4 rating scale to assess the severity of usability issues, as well as a 0 to 3 rating scale to assess how well the components allow users to adjust audio. The two scales are as follows:

• Severity Ratings for Usability
• 0 = I don’t agree that this is a usability issue at all/directly adjusts volume of audio
• 1 = Cosmetic problem only: need not be fixed unless extra time is available on project
• 2 = Minor usability problem: fixing this should be given low priority
• 3 = Major usability problem: important to fix, so should be given high priority
• 4 = Usability catastrophe: imperative to fix this before product can be released/does not control audio in any capacity


• Audio Control Ratings
• 0 = directly manipulates audio
• 1 = allows users to view audio status/volume, some potential capacity for control
• 2 = allows users to view audio status/volume, no capacity for control
• 3 = does not control audio in any capacity

User Definition

In general, we believe our findings and recommendations are generalizable for the development of an auditory cognitive aid to be used across multimodal professions. So for this study we defined different targets.

Our primary targets are astronauts who perform long-term space flight. For an astronaut, focus on the operator is imperative, as the human ability to ignore distraction or inhibit reaction to attention-grabbing stimuli decreases with fatigue, CO2 exposure, and information overload - all of which are likely to occur in prolonged space-travel.

Our secondary target users are the healthcare workers that are operating under stressful conditions.

We believe that cognitive aid can help these two target groups to perform better under stressful conditions.

For testing our prototype we decided to recruit undergrad students from NCSU.

Storyboard

We made a storyboard to see how the user is going to interact with a DR headset to reduce a sound.

Here, in this scenario, Robert and Anna are inside a spaceship. Robert sees that Anna is trying to talk with him but the loud engines of the spaceship are too distracting. He is unable to hear her clearly. So to be able to hear Anna perfectly, he needs to reduce the background noise. For doing that, he goes to the audio setting in his DR headset. The DR headset is able to detect two different noises. It also displays the current volume percentage at which Robert is able to hear the noises. He clicks the background noise and reduces it to 20% by lowering his two fingers. In the end by using the diminished reality Robert was able to focus better on his conversation with Anna.

Assesing Physical Gestures

• Originally decided to focus on using predetermined physical gestures.
• Have participants complete a reading task or watch a video.
• Mimic auditory de-emphasis over Zoom.
• Pre- and Post-Test Surveys to gather additional qualitative information.

Wizard of Oz Method

We did some research and detected the tasks our users are supposed to do to interact with the DR system, after that our team went through the cognitive walkthrough step to define the process of prototyping.

We found wizard of oz and zoom video conferencing software the best method for our user research and prototyping for 2 reasons, first of all, we consider our project as the very first steps in designing an interaction system for the DR headset so we are safe keeping it simple and lo-fi during this stage.

Secondly, the pandemic of COVID-19 halted all in-person user research so we were not able to use a holokit or a similar tool to simulate AR/DR experience.

For our wizard of oz study, we used dual monitors, one screen for sharing with the user and showing the interface to the user and one for controlling the audio distraction by the wizard that was one of the team members.

This is the display that was shared with the user during the study and mimicked the user perspective of mixed reality after wearing the DR headsets. You can see that the user is watching a video within the zoom setting.

Using the Wizard of Oz method, we intend to manipulate the audio of a distractor in response to indicators given by a participant while they complete a task. These indicators could be either physical gestures or voice commands, and the task would be watching a Ted talk video.

We decided to have participants use pre-determined physical gestures in an attempt to learn how users would feel about using gestures as a means of control. We also became interested in how well voice commands could be used to diminish a noise and volume based distractor as well. Additionally, we were curious about what potential gestures participants would come up with on their own.

These ideas greatly influenced the design of our study and helped us gather additional information concerning how users would like to interact with diminished reality and what the control system could potentially look like.

Strengths and Weaknesses of Alternate Designs

Our cognitive evaluation of the DR system began by comparing two main designs: whether the system would rely on voice-commands or physical gestures.

Strength: Since using a physical gesture is a visual stimulus and the distracting audio is auditory, it will be relatively easy for the DR system to distinguish between the different sensory sources. The use of physical gestures allows for a faster response time between the user performing the action and the system giving appropriate feedback (in this case, lowering the audio volume appropriately).

Weakness: A counterpoint to using physical gestures is that the user will first need to focus on the hand in order for the system to recognize and anticipate a command.

To further explore the use of physical gestures to control the degree to which an auditory stimulus may be diminished, we evaluated 3 alternative gestures to use as a default setting for the system:

Using two fingers was universally chosen among the designers as a way to signal the desire to change audio level. Moreover, given its simplicity, this gesture has the benefit of being able to be performed rapidly.

It was determined that gesture 3 was the better option for our proposed DR system. The use of 2 fingers allowed for fine movement that could be distinguished from any other movements the user may engage in during system use and could easily be adopted because it mimics a commonly employed action (e.g. scrolling on a touch-screen device).

This gesture however also comes with its flaws. The user may need to perform the gesture each time a reduction in audio is desired (i.e. performing gesture once causes the system to reduce audio by 4 levels. To reduce by 8 levels, gestures will need to be done twice).

Although our initial design involved the use of a predetermined gesture, during our evaluation it was agreed that a user may want to customize their system to their own liking. Not everyone may be comfortable with using a default gesture, so we have included the option of a user-created gesture for further evaluation in later parts of this project.

User Testing

The purpose of this project is to understand how users want to interact with a DR environment, specifically as it pertains to lowering a noise distraction to maintain focus on a specific task.

Prior to the experimental task, participants completed the consent form and pre-test survey in Qualtrics. In the survey participants were asked questions concerning how they interact with audio, what functions of technology they employed to decrease volume and how volume typically affects their ability to perform various cognitive tasks.

Consequently, users watched a video in 3 steps and controlled the environmental noise with their suggested gesture, our proposed gesture, and a voice command.

Following the experiment, they completed a series of post-test surveys to give feedback about their overall experience with the simulated conditions. In addition to obtaining qualitative feedback regarding our system-control design, the NASA Task Load Index was completed post-testing to assess participant perception of workload during each task condition of the experiment.

Final Design Recommandation

Keep unchanged:

• Headset design
• Head size is a factor so use generic measurements.
• Data showed user preference for gestures as method of control
• Voice commands as alternative option

Consider:

• Display changing audio level or sound perception as part of device feedback to the user.
• Remove the option of creating your own gesture.
• Gradual decrease of sound with gesture performance.
• ADA accessibility requirements:
• • Visually impaired
  • Hearing impaired
  • Physical range of motion

Mockup Design

Based on the results and storyboard, we made a mockup to show in detail how the user is interacting with audio in the DR headset; Scrolling down using both the index and middle finger.

This gesture made the most sense since it mimics scrolling when using a smartphone, and we envisioned manipulating volume bars in an established system.

Final Prototype Design

Based on results from our study, we determined that our predetermined physical gesture was seen as the best way to control audio, but DR users would benefit from a control system that was multi-modal and had some capacity for voice control as well.

Future iterations of this study:

• Run a few test-trials to receive “test feedback” from participants.
• Have participants select from some pre-designed displays in post-test survey to guide design ideation based on preference.
• This can be used to determine whether further revisions to the system need to be made.
• Pursue a larger, more varied sample pool and more consideration to time constraints.
• Once an actual device prototype is available, additional investigations into how the controls affect task performance and other cognitive functions should be conducted to assess whether further revisions to the proposed system are warranted.