Key Insights from Cycle 2 Testing Debrief Sessions
As a reminder, the key Cycle 2 research question was: To what extent and in what ways are visitors using the app (before, during, and after their visit)? Sub-questions included: 1) What features do visitors use and when do they use them? 2) What were visitors’ goals for using a specific feature (motivation)? 3) Who engages with the app? How did they use the feature of the app (e.g., to show their children something? To get more information about something? etc.) and 4) What are visitors’ perceptions of the value and usefulness of the app?
Summary of Cycle 2 Testing
Use of Features and Content
- Before the visit, the families who looked at the app tended to browse the features just to familiarize themselves with it. A few caregivers at Sciencenter used the age filters to see content appropriate for their children. At OMSI, the “Map” and “Info” were used most often before the visit.
- During the visit, “Try This” was used by visitors across sites. In the case of younger children (<6 years old), caregivers usually read the prompts and then used them with their children to facilitate the interaction with the exhibit. In the case of older children (>8 years old), the children tended to use the app themselves and show it to the adults with them.
- Families were intrigued by the Camera feature, yet had many questions about it. They did not know exactly what it did and what to use it for. They also had privacy concerns about where the photos would be posted and who had access to them. Sciencenter and MLS reported that because there were no posts under the Camera feature made it especially confusing. They suspected that having it populated would help respondents understand how the feature works.
- “What Do I Do?” was also used at OMSI and CMH during the visit. The OMSI exhibit did not have signage, so this feature was more applicable for that exhibit than other sites that had signage at their exhibits. At CMH, visitors appreciated when the instructions on app were more thorough than the signage and felt exhibit instructions on the app were more convenient than reading a sign.
- At CMH families tended to stick with the initial features they looked at and not explore other features. There are indications that this was because they were not familiar with the app ahead of time. This raises the question of whether it’s realistic that the majority of families will download the app and explore it prior visit. If the actual expectation is that many will download on-site, then it raises the question of how intuitive and/or intriguing the categories are.
- One additional type of content that participants specifically asked for were more “fun facts” related to exhibit components that they could quickly share with their children. For instance, in CMH’s Kidtropolis Vet Clinic a caregiver wanted short facts about health/anatomy of the animals featured. Likewise, at the job center, caregivers asked for interesting facts about the different professions featured there.
- At least one site did not see audio and video being used during museum visits. They suspected that the families did not notice that it was an option. Other sites had minimal video and audio and, as a result, did not see them being used.
- At CMH, the Map feature of the app was regarded as a more convenient way to access a map than a large map printed on paper.
- Partners had little to no data from families regarding their use of the app after their visit. Sciencenter respondents said that they were interested in using “Research Says” and “More Resources” after their visit, as opposed to during their visit.
Effect on Behavior with App
- There was evidence that “Try This” helped families interact with the exhibitions in deeper ways. At OMSI, families were observed trying all the options described by the app a component, as opposed to trying one way of using the component and moving on. “What Do I Do?” also changed interactions so that visitors did not just give up on a component without out how to use it. Sciencenter had respondents that said “Try This reminded them of areas that their families had forgotten about (for example, the blocks). It also allowed them to learn new ways of interacting with favorite places (for example, the water). At CMH, caregivers reported that the app prompted them to create new types of questions for their children than they would normally ask. These people would start with the topic or questions the app provided, then use this information as a jumping off point to formulate their own questions for their children.
- The team continued to see a tension for caregivers of younger children between using the app and engaging with their children. Caregivers did not want the app to get in the way of the families’ interaction together at the exhibit and wanted to make sure the children were nearby, safe, and interacting with the physical exhibition. Some caregivers were able to integrate the app into the experience by using such techniques as reading the content while their children were engaged in the activity before talking to them about it or surreptitiously reading the content off the phone.
- Some visitors were observed tapping photos, expecting something to happen. In one case, tapping on a photo caused the photo to open in another program outside the app.
Other Issues to Keep in Mind
- Cycle 2 was usually completed by respondents who were familiar with the museum. We suspect that first-time visitors will use the app in different ways than long-time members. Is this an issue? Will people use the app on subsequent visits? How could the app be designed to prompt use over time and provide a compelling experience for both new and returning visitors?
Cycle 3 Focus & Partner Sites
- For this final cycle, we recommend that CMH, OMSI, and Sciencenter focus their testing on comparing visitor interactions at one or two exhibits with and without the use of the app. There are three reasons for this:
- The primary focus of the summative evaluation will be a comparison study of visitors with and without apps at select exhibits. So it’s critical to make sure you have the best possible developed for summative evaluation. Otherwise, we will likely not see significant differences with and without the app.
- We are seeing some promising interactions with visitors using exhibits in different ways due to the app. It merits further inquiry in cycle 3 so it can inform final development prior to summative.
- The summative will focus on one exhibit each at OMSI and Sciencenter and a few at CMH; we need to start narrowing the focus in cycle 3 to be prepared for summative.
Note: Ideally, a little more content should be added to Sciencenter’s app so that visitors are not tripped up seeing only one option under “Try This”. However, if that is not possible at this point, the with/without app testing can proceed with the understanding that the extra page may continue to be an issue with visitors.
- We recommend that NYSCI and MLS continue to focus on testing the same questions as in Cycle 2.
Cycle 3 Research Questions
|Site||Focus||Key Research Questions||Sub-Questions|
|Cycle 3: Interactions at one or two exhibits with and without app||How does XCL affect visitor interactions at the exhibit and how do these interactions differ when visitors are not using XCL?
|· What are families’ perceptions of the quality of their exhibit experience with and without using XCL?
· To what extent does using XCL positively impact visitors’ knowledge and/or skills and does this differ from families who do not use it?
|Cycle 3: Behaviors||To what extent and in what ways are visitors using the app (before, during, and after their visit)?
|What features do visitors use and when do they use them?
What were visitors’ goals for using the specific feature (motivation)?
Who engages with the app? How did they use the features of the app (e.g., to show their children something? To get more information about something? Etc.)
What are visitors’ perceptions of the value and usefulness of the app?