Joel Calumpong

CLIENT

PROBLEM SPACE

• Textbooks are expensive, and students and faculty may only use a small portion of them.

• Postsecondary faculty receive no training in how to instruct.

• Postsecondary faculty are not very tech-savvy and have trouble using existing course creation software.

BUSINESS NEED

The ability to create courses on the platform is essential to Argos’ business model, and they did not have an existing experience that met or exceeded existing market expectations.

CHALLENGE

In 10 weeks (Jan 2022 - March 2022):

“Create an intuitive and effective way for creators to set up course goals, outline the course, prototype units, add content, fine-tune the design and test it with early adopters”

OUTCOME

My team and I created a validated course design experience that incorporated the client’s unique value proposition as a first-to-market feature set.

“I really like how you thought about the platform as a tool for professional development. You’ve really captured the way we like to think about it…” - Argos’ client team

MY ROLE

I worked in a team of 4 as co-facilitator. My contributions are detailed in the project’s phases below.

Process

Client Kickoff

The Argos’ client team discussed their existing product and intended market.

The team learned Argos is targeting 2 types of users: postsecondary faculty and instructional designers.

Instructional designer create courses using learning science which are validated best practices that maximize learning outcomes.

We also learned the following which informed the product design criteria:

Client Product Goals:

• “We want [our product] to make teachers better.”

• “….provide scaffolding for educators.”

• “…learning from the ‘book’ & class to be more connected.”

When asked “How will we know if this project is a success?” the client responded: “If you can teach us something new.”

Comparative Analysis

We performed a comparative analysis and heuristic analysis with the goals of:

1. Define the minimum feature set needed to compete in the courseware space

2. Gain an understanding of the general usability involved in current course creation flows

3. Map the existing landscape to understand where Argos competes

Challenges:

All the competitor products required university affiliation for access. Requests for demonstrations went unanswered.

Findings:

No one is making the learning science easily accessible

Menu and “course” nav on left, editing menus in top of right

Material organized into modules/units with formative and summative assessments

Empathy Interviews

The team interviewed 9 participants representing a mix faculty/instructional designers.

The competitive analysis helped inform the interview script regarding asking about specific features and how tasks are accomplished.

a major Discovery

We uncovered an insight which challenged one of our client’s user-behavioral assumptions:

Our client was stunned. Part of their roadmap is to use quantitative data to drive course design improvement.

“I use the pain of the previous class to inform the design of the next.”
Faculty Member: Professor of Linguistics / Male / Early 60s

“I would rather just work on developing relationships… and use my ability and intuition as an instructor.”
Faculty/Instructional Designer: Associate Director, Learning Experience Design / Female / 46

The finding informed us that course design is influenced by an iterative, qualitative data feedback loop with learner cohorts.

Narrowing down opportunities

These three opportunities most closely addressed Argos’ value proposition and the product goals we elicited from our client meetings.

Ideation

During our research synthesis phase, we crafted personas to help guide and assess the team’s concepts and ideas.

This was helpful to keep in mind as the two personas have very different levels of comfort with using technology and learning science.

After agreeing on a master flow for course creation, each team member sketched ideas for different parts of the experience.

Prototyping/Usability Testing

The team worked together to build out a prototype that incorporated an early version of a style tile so we could also test look and feel with our participants. We built out the sub-flows that would test critical assumptions and dependencies for our designs.

Methodology

Our testing plan centered around 3 main questions (reasonings are below):

1. Will participants understand drag and drop as a way to assign content?

   • If successful, we planned to reuse this pattern across the platform as a means to make the interface more intuitive and user-friendly.

2. Will participants understand the course design analytics feature?

   • As a first-to-market feature, we wanted to gather “task- success” data as a benchmark for later development by asking if participants could explain the meaning of the displays or find the explanations on the page.

3. Will participants find the course design analytics feature useful?

   • We felt it was critical to get measures on this since it directly touches upon Argos’ value proposition.

Results: Areas of Success ***(REPLACE IMGS W/ Higher res)

Results: Opportunities for Improvement ***(REPLACE IMGS W/ Higher res)

Final Prototype ***(replace gifs w/ vid clips)

“Help instructors be their best selves through learning science data”

We kept this in mind when moving forward with the reimagining of Sojourner. Learning science is baked into the product meaning that the very act of creation is a realization of user interface enabling course creators to do better than they might’ve imagined.

Course Creation: One screen sets you up for success
Just one setup screen sets up the course and its entire structure. Access this part of the prototype here.

Course Structure: A million moving parts all in one place
A populate-as-you-go outline of the course allows course creators to see what's been planned at any given time in the course. Using data from the course creation screen, the overall structure of the course is built out for immediate slotting of content.

Access this part of the prototype here.

Drag & Drop Content: Set it and forget it

All content can be planned and assigned using a drag and drop interface at many levels in the interface. The UI gives feedback about where your content will land as well as a confirmation message that assignment was successful so that less tech savvy users understand what they're doing and have accomplished.

Access this part of the prototype here.

Design Analytics: Insight on how to Level Up Your Course

Course creators can leverage the power of Sojourner to analyze and give insights into what the student experience might be like through summary dashboards that give a high-level view of how the course content aligns with learning science considerations.

Summaries can be expanded to provide a module-by-module view for those wanting more granular information.

Access this part of the prototype here.

Course creators can also receive insights concerning how the course composition breaks down regarding task type, grade weighting, and learning type as well as how course content measures against standards of accessibility, equity & inclusion, personalization, and feedback. All of those listed areas are important to learning science practices and helping course creators realize if the course matches their vision.

Much like the general course structure, the learning objectives table gives a summary of the structure of learning goals and how they are supported. Hover state pop-ups help keep the table readable while giving more specific information.

The entire happy path final prototype can be accessed here.

Client/Impact

Since the client asked us to completely reimagine the course creation experience, we didn't have baseline KPIs to quantify how successful we were. However, the client feedback after the case study presentation and final demo were as follows:

“Was it a success? Overwhelmingly so. Every week we were excited to see what you’d come to us with.”

"You’ve definitely expanded our thinking about something we’ve been struggling with."

"You have definitely taught us something new"

"In all seriousness, if we had more funding, we’d be offering you on-staff positions"

Next Steps

The 10-week challenge my team and I tackled represents a small piece of Argos’ vision of a creator economy platform for course creators. Further cycles in the UX process would focus on fine-tuning the new course creation flow while considering how this flow interlocks with other flows that form the ecosystem of the platform.

Research Actions:

1. more empathy interviews and usability testing sessions with “pure” faculty-type users

   • to learn how to position and talk about learning science as a means to drive greater adoption

2. interview a wider variety of subject matter course creators

   • to see how subject matter may or may not affect the course creation process

3. test the different flows of the most up-to-date prototype with more complex tasks and measure SUS

   • to establish benchmarks to better track usability over iteration cycles

Design Actions:

1. create the “find & adopt” path of course creation

   • a few faculty members said they would want to “get the lay of the land” or normally work off an existing course

2. define what the OER selection and other content selection experiences could be

   • use of OER is critical to Argos’ plan to supply content while minimizing student costs

3. tweak or alter the course creation experience to connect with other planned user task flows

   • to start supporting longer user journeys and move closer towards fully realizing the monetization model

Lessons Learned

PUT YOUR CLIENT ON SPEED DIAL
The weekly touch bases with the client proved essential in keeping the project in line with their business objectives while also helping to close the iteration loop faster -thereby accelerating the development process.

BEING STRATEGIC ABOUT USER RESEARCH IS A MUST
The time with users is never enough so it was vital to understand how our user research would help address the business goals of the product. This was especially important to consider since the team had to do so much generative research to build out product criteria for an experience that didn’t exist.

CLIENTS HAVE ASSUMPTIONS EVEN IF THEY ASSUME THEY DON’T
When the team kicked off the project, the client briefed us with their understanding of who course creators are and what they do. While a lot of what they communicated to us seemed to be validated in our research, there were some important aspects that were invalidated in the same body of work. The takeaway is: Trust, but verify.