How to Run a Learning Technology Pilot That Produces a Clear Decision

A learning technology pilot should help an institution make a decision. It is not simply a smaller version of a full launch, and it is not successful merely because the technology operated. A useful pilot begins with a clear question, a bounded use case, and agreement about what evidence will inform the next step.

The start of an academic year can be a practical time to identify needs, but timing should follow staff and learner capacity. A pilot that competes with essential term-start work may produce rushed participation rather than useful learning.

Define the decision first

Write the decision the pilot will support. Examples might include whether a browser-based orientation should move to a wider group, whether a virtual interpretation format fits a type of collection, or whether staff can maintain the content responsibly.

Then define possible outcomes:

• proceed to a broader implementation;
• revise the approach and test again;
• keep the solution for a narrow use case;
• stop because the need, fit, or operating model is not clear.

A no-go decision can be valuable when it is based on agreed evidence. It prevents an institution from continuing only because work has already begun.

Keep the scope specific

A pilot should include enough of the real experience to test the important assumptions. It does not need to represent every audience, location, or feature. Choose one module, one learner journey, or one operational process.

Document what is inside and outside scope. Specify the content, audience, delivery setting, supported devices, accessibility requirements, and review rounds. If new ideas arise, record them for later rather than expanding the pilot until its purpose becomes unclear.

Agree on success criteria upfront

Success criteria should connect to the decision. Avoid vague goals such as “engagement” without defining what will be observed. Depending on the use case, a team might examine whether learners can complete key tasks, whether instructions are understood, whether staff can support delivery, and whether the experience works on intended devices.

Useful evidence can include:

• task completion and points of confusion;
• learner and facilitator feedback tied to specific activities;
• accessibility issues and the effectiveness of alternatives;
• technical support requests and device constraints;
• staff time required to review, update, and deliver content;
• content accuracy and approval findings.

Collect only what the decision requires. Respect privacy and explain to participants how feedback will be used.

Bring the right stakeholders together

A pilot often crosses academic, operational, and technical responsibilities. Identify a sponsor who owns the decision and a day-to-day lead who coordinates the work. Include the people responsible for teaching or interpretation, accessibility, technology support, communications, facilities or collections, and privacy where relevant.

Clarify who can approve content, who can request changes, and who decides when the pilot is ready for learners. Without this structure, review can become a series of conflicting preferences.

Learners or visitors should be involved early enough to influence the experience. Testing only after everything is polished makes meaningful changes harder.

Use a timeline with decision points

A practical pilot timeline includes:

1. Scope: confirm the question, audience, responsibilities, and constraints.
2. Content: gather source material and approve the learning outline.
3. Build: produce the smallest complete experience that tests the assumptions.
4. Review: check accuracy, accessibility, usability, and technical delivery.
5. Use: run the pilot in the intended setting with support available.
6. Decide: compare evidence with the agreed criteria and document the outcome.

Leave time between review and use. Changes that affect navigation, captions, or content accuracy need proper checking.

Measure the whole operating model

A visually strong module can still be a poor institutional fit if updates are difficult, ownership is unclear, or learners cannot access it reliably. Evaluate production and maintenance alongside the learner experience.

Ask who will keep links current, update service information, approve new media, answer support questions, and archive older versions. Consider how the module fits existing orientation, courses, exhibitions, or staff workflows. The pilot should reveal these responsibilities rather than hide them.

Close with a documented decision

At the end, summarize what was tested, what was learned, what remains unknown, and which criteria were met. Separate required changes from optional enhancements. Record the decision, owner, and next review point.

For guided virtual learning, a scoped pilot can test the full connection between place, content, learner guidance, and institutional operations. The aim is not to prove that immersion is impressive. It is to determine whether a specific module is useful, accessible, maintainable, and appropriate for the institution’s purpose.