Holly Champion: Learning for Life
BEST Buddies Programme
Implementation & Evaluation
Roll out plan
LMS Considerations
The company's Learning Management System (LMS) is Absorb. It is primarily used for hosting e-Learning content and tracking completions, though it includes functionality to schedule and track instructor-led courses as well. Microsoft Teams is used to host virtual classrooms.
​
Most of what is required for this course is available through the company's Absorb package. Gaps will be plugged using applications from the Microsoft Office suite:
Required functionality
Available through Absorb?
Alternative source
Notes
e-Learning requirements
Ability to host e-Learning content
Yes
N/A
Ability to structure a learner's journey through the course
Yes
N/A
Ability to assign e-Learning to learners
Yes
N/A
Ability to set due dates for e-Learning completion
Yes
N/A
Virtual classroom requirements
Ability to schedule virtual classrooms
Yes
MS Outlook
Outlook is currently preferred by the business
Ability to host virtual classrooms
No
MS Teams
Ability to host recordings of virtual classrooms
Partial
MS SharePoint
SharePoint is currently preferred by the business
Ability to record attendance of virtual classrooms
Yes
N/A
Assessment requirements
Ability to host graded assessments (multiple choice)
Yes
N/A
Ability to set the number of attempts a learner may take to pass an assessment
Yes
N/A
Ability for learners to upload coursework
Yes
N/A
Ability for assessors to access uploaded coursework for marking
Yes
N/A
Ability to record the score for uploaded coursework
Yes
N/A
Learning journey requirements
Ability to set pre-requisites (e.g. preventing a learner from accessing advanced content before they have completed the foundation content)
Yes
N/A
Ability to apply branching to the learning journey (e.g. setting a pre-test at the start of a course/module and allowing the learner to bypass the content if they pass, or automatically enrolling the learner into remedial/refresher content if they fail a course)
Yes
N/A
Reporting requirements
Ability to track and report on learner progress through e-Learning
Yes
N/A
Business preference is to export the raw data to Excel, where it can be analysed before distribution
Ability to report on attendance of virtual classrooms
Yes
N/A
Business preference is to export the raw data to Excel, where it can be analysed before distribution
Ability to track and report on individual assessment scores
Yes
N/A
Business preference is to export the raw data to Excel, where it can be analysed before distribution
Ability to produce granular reporting of learner assessment responses
Yes
N/A
Assessments must be built in Absorb (not Articulate or other authoring tools) to produce these reports
Business preference is to export the raw data to Excel, where it can be analysed before distribution
Ability to report on coursework scores
Yes
N/A
Business preference is to export the raw data to Excel, where it can be analysed before distribution
Ability to report the overall score per learner for the course
Partial
N/A
Coursework score is excluded from the report. Individual assessment and coursework reports must be combined in the Excel report template to produce the overall score
Communication requirements
Ability to notify learners that they have been enrolled in a course
Yes
N/A
Ability to schedule nudges for learners to remind them that they are enrolled in a course that they have not completed
Yes
N/A
Ability to send learners confirmation that they have completed all requirements of a course/module
Yes
N/A
Ability to create course/cohort specific forums
Yes
N/A
Microsoft Teams is preferred as it does not require additional log in to access and is open as part of the learner’s normal working day.
Learner requirements
A personalised learner dashboard
Yes
N/A
Ability for learners to easily identify courses that they have been enrolled in
Yes
N/A
Ability for learners to view the date they completed a course and the score they got on the related assessment
Yes
N/A
Ability for learners to access certificates they have earned
Yes
N/A
Ability for learners to easily identify which courses they have been enrolled in are still to be completed
Yes
N/A
Manager requirements
Ability for managers to view their team members’ progress through assigned learning content
Yes
N/A
The implementation plan
This plan provides an overview of the roll out and evaluation timeline for the BEST Buddies programme. This is broken down into 5 connected phases (shown below and colour-coded for ease of reference). More detail, including personnel resourcing requirements, can be found below. (NB: names other than my own have been redacted. Placeholder text remains in their place.)
This task list, along with an estimate of the number of working hours required to complete each task (not shown, but available via the ‘FTE Estimate’ tab of the plan spreadsheet) has been used to generate a projection of the FTE (full time equivalent) resource required throughout the implementation and initial rollout of the project, as can be seen below.
Resource requirements
This section focuses on requirements other than personnel. For information about FTE and personnel requirements, see the implementation plan in the previous section.
Learners
Learners will not be expected to access or use any hardware or software that is not part of the standard setup for a member of staff:
​
-
Company laptop
-
Microsoft 365 Suite
-
Headset with microphone
-
Learner access to the Absorb LMS
-
Internet connection
Templates that learners are expected to use within the programme will be accessible via download from the e-Learning modules and from the dedicated course folder on the Academy Hub.
Facilitators
In addition to the standard equipment and software common to all staff, facilitators will also need Learning Administrator privileges in the Absorb LMS. This includes the ability to:
​
-
Assign content to staff through Absorb
-
Generate activity and assessment reports
-
Review and score learner submissions
Comms plan
Pre-launch comms
These communications will ensure that managers are fully briefed on the course ahead of the launch. They will provide a high-level overview of the course structure and purpose (including the learning outcomes), as well as the anticipated benefits of staff completing it, and will allow managers to nominate members of their team for inclusion in the pilot cohort.
​
Comms will be delivered through a slot in the monthly Learning & Development Working Group meeting and through an article in the manager comms. The initial comms should be delivered no more than 2 months before the expected launch date. Updates will be delivered by email to the management group or through the manager comms.
Suggested wording for the initial comms
Introducing the Academy's BEST Buddies Programme
​
In <month> the Academy will be launching its BEST Buddies Programme, which is designed to help staff organise, deliver and evidence effective on-the-job training to their colleagues. Over the 6-week programme, attendees will learn how to:
​
-
Generate specific, measurable, achievable, relevant and time-bound (SMART) learning outcomes for any process they are required to train
-
Prepare a training plan to structure the training delivery of any process they are required to train
-
Identify evidence that a learner has achieved a specified learning outcome and compile this evidence into a proof-of-learning record (POL) for the learner which demonstrates that they are ready to progress to "Gaining Licence" for the specified skill
-
Modify their delivery to suit the learner they are supporting whilst maintaining the standards required to achieve the learning outcomes
-
Use the tools and templates provided by the Academy to ensure a consistent approach to on-the-job training across the company
-
​
Successful implementation of this programme is expected to bring a wide range of benefits for the company including a reduction in errors that result in breaches, as well as a growing library of training plans that Buddies in your team can use to train their less-experienced colleagues.
​
The course is primarily aimed at staff in zone 2 of the P&P scheme who wish to progress to zone 3 and will result in a portfolio of evidence which they can use to demonstrate their competence against the upgrade requirement.
​
If you would like more information, or would like to nominate a member of your team to take part in the pilot, please contact the Academy at <Academy mailbox>.
Follow-up/update comms
The nature and content of these comms will depend on circumstances, but it is recommended that a reminder about the coming launch is included in each manager comms after the initial announcement until the launch date as a standing item.
​
The following wording is suggested:
Have you heard about the BEST Buddies Programme?
​
Click here <insert hyperlink to original announcement> to see what it’s all about, or contact the Academy for more information or to nominate a member of your team.
Pre-launch comms
These comms will be targeted at all staff through the following channels:
​
-
Viva Engage (announcement)
-
CEO’s Message (announcement)
-
Announcement on the Academy Hub (announcement & info page)
-
Team huddles (announcement)
-
The Big/Wee Tune In (schedule dependent) (presentation)
Suggested wording for the announcements
Can YOU be a BEST Buddy?
​
Do you want to step up to progress in your career?
​
Do you want to tune in to learn how to effectively share your knowledge and experience?
​
Do you want to be a force for good by helping to improve the quality of your team’s work?
​
If you answered YES to any of these questions, then the Academy’s BEST Buddies Programme can help you!
​
Click here <insert link to info page on Academy Hub> to find out more!
Academy Hub info page
This should provide high-level information for potential learners and their managers to find out more about the course before taking the decision to sign up. The following information must be included:
​
-
Course purpose, including learning outcomes
-
Who the course is suitable for (links to the P&P scheme)
-
Scheduling (how frequently the course will run, how each cohort will be scheduled and the learner’s expected time commitment)
-
How the course will be assessed
-
How to sign up
-
Who to contact for more information
-
Testimonials from previous cohorts (won’t be available until at least one cohort has completed)
-
Welcome/endorsement message from CEO (desirable)
Tune In presentation
Exact content will be determined by the slot secured within the event, but should include at a minimum:
​
-
Course purpose, including learning outcomes
-
Who the course is suitable for
-
How to find out more/sign up
The learning agreement
To ensure that learners and their managers have a clear understanding of the time commitment required for completing this course, a learning agreement is proposed. Such an agreement will help to protect both the learner and their manager by helping to ensure that the impact on scheduling is taken into account before the learner begins the course. In turn, this will help to mitigate the risk to the Academy’s schedule of learners being pulled from the training at short notice or requiring extensions to the deadline because of scheduling requirements.
​
It is recommended that this confirmation be obtained electronically (e.g. using PowerApps) and then a copy of the agreement 'signed' by all parties is sent to each 'signatory' for their records so that both the learner and their manager have easy access to the information. A learner will not be enrolled into the course until all three parties have signed the agreement.
​
The following wording is suggested:
Learner's name: ______________________
​
By taking part in the BEST Buddies Programme, I am committing completing the following modules and submitting the required coursework by <course deadline>.
​
I understand that 40% of my final score is derived from the module assessments and the remaining 60% is derived from my coursework.
​
The modules that must be completed by this deadline are:
-
Defining Your Goal
-
Planning For Success
-
Let the Evidence Show
-
Meet Your Learner Where They Are
​
I understand that the course will take up to 12.5 hours to complete and that my manager has agreed to schedule time for me to complete this work during my normal working hours. I understand and accept that I am responsible for managing my progress through the course tasks for the duration of the course and that it is my responsibility to notify my manager and the Academy as soon as possible if I believe that I will be unable to complete the required work within the scheduled time and/or by the course deadline. I understand and accept that reasonable and timely requests for additional learning time within my working hours will be considered in light of the needs of the business, and that it may not be possible to grant such requests. I will be able to take holiday and flex time as usual during the course.
​
I understand that attending the virtual classrooms is optional, though recommended, and that not attending will not be counted against me in my final score.
Learner signature: ________________________________ Date: _________________________
​
Manager's name: ___________________________
​
By authorising <learner name> to take part in the BEST Buddies Programme between <course start date> and <course end date>, I confirm the following:
​
-
<Learner name> will need up to 12.5 hours of learning time to complete the course. This time is to be scheduled during <learner name>'s normal working hours, based on the course timeline provided by the Academy.
-
I will consider any requests for additional learning time to complete this course from <learner name> that are reasonable and timely. Each such request will be considered on its own merits and in light of the needs of the business.
-
I will notify the Academy as soon as possible if <learner name> is off sick during the course and that the sickness is expected to last more than 1 week. When <learner name> is off sick for more than one week during the course, I understand that will have reduced the amount of time they have to complete the course and will make reasonable accommodations to ensure that <learner name> is not disadvantaged. I understand that this may require them being transferred to a later cohort.
-
I will notify the Academy as soon as possible if <learner name> leaves my team and/or the business prior to or during the course.
Manager signature: __________________________ Date: __________________
Academy Manager: _________________________
​
On behalf of the Academy, I confirm that <learner name> has been accepted onto the BEST Buddies Programme running from <course start date> to <course end date>.
​
The lead trainer for this cohort is <lead trainer name>, who can be contacted via Teams chat or by email to <Academy mailbox>.
​
<Learner name> will receive joining instructions no later than <joining instructions date>.
Academy Manager signature: ____________________ Date: ________________
Enrolment comms
The following comms will be sent to the learner as part of their enrolment:
​
-
System-generated comms (automatically generated when the learner is enrolled via Absorb)
-
Virtual classroom joining instructions (sent in the body of the Outlook invite)
-
Course welcome pack
​
With the exception of the course welcome pack, these are based on existing templates.
System-generated comms
Dear <learner name>,
​
You have been enrolled into the BEST Buddies Programme, which includes the following e-Learning modules:
​
-
Defining Your Goal
-
Planning For Success
-
Let the Evidence Show…
-
Meet Your Learners Where They Are
​
You can access these modules via the ‘Mandatory Courses’ section of your Absorb dashboard <URL for Absorb dashboard>.
Virtual classroom joining instructions
Course introduction
We’re thrilled that you have joined the BEST Buddies Programme. You are invited to join this online session, which will formally welcome you onto the course and explain how the course will run.
​
It’s designed to give you the information you need to get started and give you the opportunity to ask any questions you have at this stage.
​
We look forward to seeing you there!
Virtual classroom 1
You are invited to join this virtual classroom, which will allow you to explore the principles covered in modules 1 & 2 of the BEST Buddies programme in more detail.
​
To get the most out of the session, it is recommended that you complete the following modules before you attend:
​
-
Defining Your Goal
-
Planning For Success
You can find these modules via the ‘Mandatory Courses’ section of your Absorb dashboard.
Virtual classroom 2
You are invited to join this virtual classroom, which will allow you to explore the principles covered in modules 3 & 4 of the BEST Buddies programme in more detail.
​
To get the most out of the session, it is recommended that you complete the following modules before you attend:
​
-
Let the Evidence Show…
-
Meet Your Learner Where They Are
You can find these modules via the ‘Mandatory Courses’ section of your Absorb dashboard.
Course graduation
To be sent after the completion deadline. To be sent to both the learner and their manager.
Well done for making it this far!
​
We’d love for you to join us at the graduation party, which is an opportunity for us to showcase some of the highlights of your journey and to celebrate your cohort’s success.
Course welcome pack
The welcome pack will be sent to the learner after the course introduction session and will contain the following information:
​
-
Welcome to the course from CEO
-
Cohort schedule
-
Lead trainer’s contact details
-
Assessment brief
-
Links to the e-Learning modules
-
Assessment rubric
-
A copy of the learning agreement
​
It is still to be decided whether this will only be sent by email, or if it will also be published on Absorb (excluding the individual’s learning agreement) as a non-mandatory module of the course.
The assessment rubric
​​​The assessment rubric will be sent to each learner as part of the learner welcome pack. Part of the optional welcome session will also be dedicated to explaining the rubric to the learners. This will help to set learner expectations early in the course, as well as providing them with useful information about how to present and submit their coursework.
Completion comms
These are based on existing templates.
e-Learning module complete
Absorb automatically sends the learner a system-generated email informing them that they have completed a module. It confirms the module completed and the completion date. The template can be edited, though this is not normal business practice, as these notifications generally get deleted unread.
Virtual classroom next steps
Manually created email sent to learners after a virtual classroom which:
​
-
Thanks them for attending
-
Provides access to course handouts (where applicable)
-
Provides access to the session recording (where applicable)
-
Provides access to further resources including subject-specific contact details (where applicable)
-
Lists next steps (where applicable)
-
Requests feedback via the Academy’s Instructor-led Course Feedback Form (includes a reminder of the course name, session name and who facilitated/delivered the session).
​
It is preferred that these emails are written fresh after each session, rather than being scripted, as this makes the message feel more like it is coming from human beings rather than an impersonal entity.
Notification of course completion
This will provide the learner with their final score and feedback. This will be manually generated for each learner and sent to them before the graduation ceremony. It will include:
​
-
Their score
-
Their grade
-
A detailed breakdown of their score against the assessment rubric
-
Individual feedback
-
Details of how to access their certificate on Absorb
-
An invitation to provide feedback on the course as a whole via the Academy’s Instructor-led Course Feedback Form (includes a reminder of the course name, session name and the name of the lead trainer for the course).
Celebrating success (on-going comms)
Graduation day posts
A group photo of the successful delegates will be taken via Teams during the graduation ceremony and posted on Viva Engage with the following text:
Meet your new BEST Buddies!
​
Huge congratulations to the members of Customer Care who have recently demonstrated that they are a force for good by successfully completing the BEST Buddies Programme.
​
<list the names of the successful learners>
​
Could you be next? Click here <insert link to Academy Hub info page> to find out.
Monthly CEO update
Details of the successful learners will be included in the next available issue of the CEO's monthly update after the graduation ceremony. As well as celebrating the success of the specific cohort, this will also serve as a reminder to the business that the course is available. The following wording is suggested:
Meet your new BEST Buddies!
​
Huge congratulations to the members of Customer Care who have recently demonstrated that they are a force for good by successfully completing the BEST Buddies Programme.
​
<list the names of the successful learners>
​
Could you be next? Click here <insert link to Academy Hub info page> to find out.
Learner/manager testimonials
These will be collated from the evaluation process (see below) and added to the info page on the Academy Hub so that potential delegates can see what their colleagues have said about the course.
Evaluating the course
Evaluations are a key part of the ongoing maintenance of courses managed by the Customer Care Academy. Every course will undergo cyclical reviews throughout its life to ensure that the content remains accurate and that the course continues to provide an effective and appropriate learning experience and meets the needs of the business. As with any course managed by the Academy, high-level statistics (including learner evaluations) will be reported monthly, and the course will be subject to the cyclical review process for the duration of its life.
​
The following check points are typical during the life of a course:
Check point
Timing & frequency
Purpose
Initial needs analysis
Once only
Carried out at the start of the project
To identify the business need and target audience and to provide data to support the training recommendation.
Transition to BAU analysis
Once only
At the end of the pilot phase OR at the end of the change-related roll out
For training created to support a change
To identify whether there is a business need to continue to offer the course and, if so to confirm the target audience and what that need is. If necessary, this will result in rewriting (part of) the course to remove change-specific elements and make it suitable for the BAU business need and target audience.
​
For pilot roll outs
To review the success/failure of the pilot, including how well it met the business need and how it was received by the target audience. Where the course was deemed successful by this review, it will be signed off into BAU and will move to the cyclical review framework.
Cylical review
Annually
To review the accuracy of the content and make changes indicated by analysis of completion, assessment and feedback data. To review the on-going business need and to confirm that the target audience and training solution remain appropriate.
Monthly review
Monthly
To monitor trends in completion, assessment and feedback data allowing swift response to any undesired trends that emerge.
Ad hoc review
As needed
To review the content and (less commonly) the target audience following a change in regulations, business procedures, etc. to ensure the content remains accurate and appropriate.
There are three broad categories of evaluation that will be applied to the roll out of the course based on the nature of the data: course statistics, feedback and business KPIs.
Course statistics
The following reports will be used to monitor learner progress and success and will form part of the monthly review:
Report
Type
What it shows
Course completions
Generated from the LMS
How many learners have been enrolled in each module, how many are currently in progress and how many have completed. It also shows how much time each learner has spent on the module.
Assessment summary
Question summary
Course uploads
Academy e-Learning report
Learning & Development report
Generated from the LMS
Generated from the LMS
How many learners have been enrolled in the assessment, how many are currently in progress, how many have completed, their score and how many attempts they have made.
Lists all of the questions in the multiple-choice assessment, which options were correct, and shows how many learners selected each option.
Generated from the LMS
Shows what coursework has been uploaded by the learner and the score it received.
Excel template
Compiles the data from the system-generated reports for all e-Learning managed by the Academy, along with initial learner feedback (see below).
Excel template
Compiles the data from the session register for virtual classrooms and the data from the Academy e-Learning Report to show development time for the company.
The following reports will be used to carry out the transition to BAU analysis, as well as cyclical and ad hoc reviews:
Report
Type
What it shows
Course completions
Generated from the LMS
How many learners have been enrolled in each module, how many are currently in progress and how many have completed. It also shows how much time each learner has spent on the module.
Assessment summary
Generated from the LMS
How many learners have been enrolled in the assessment, how many are currently in progress, how many have completed, their score and how many attempts they have made.
Question summary
Generated from the LMS
Lists all of the questions in the multiple-choice assessment, which options were correct, and shows how many learners selected each option.
Course uploads
Generated from the LMS
Shows what coursework has been uploaded by the learner and the score it received.
Shows every response to every question in the multiple-choice assessment by every learner as well as what the correct response was.
Generated from the LMS
Answers report
Shows enrolments in and attendance of virtual classroom sessions.
Generated from the LMS
ILC completions
Academy e-Learning report
Excel template
Compiles the data from the system-generated reports for all e-Learning managed by the Academy, along with initial learner feedback (see below).
Feedback evaluations
Learners will be invited to provide feedback about the course in each of the following circumstances:
​
-
Immediately after completing an e-Learning module/virtual classroom
-
On completing the course (immediately after receiving their final score and feedback)
-
3 months after completing the course
Managers will be invited to provide feedback about the impact on their team 3 months after the learner graduates.
Post-module evaluations
The opportunity to provide feedback about the module or session they have just completed will be offered to the learners at the end of each e-Learning module (links are included to the form in the final part of every module) and in the follow up email after each virtual classroom session. Feedback collected at this stage is predominantly technical and based on the learner’s immediate ‘gut feeling’ about the module they have just completed. It is a useful guide to review the suitability of the course’s structure, navigation and communication. By default, feedback is anonymous, though learners have the option of providing their contact details if they wish to.
Instructor-led course feedback form
Form wording
Please take a couple of minutes to complete this feedback based on your training experience. We appreciate your openness and honesty – your feedback will help us to improve the content that we offer and the style of our delivery. By default, your response is anonymous. 1.Which course are you providing feedback for? (Please select the course from the list). 2.On what date did you attend? (If this was a multi-day event, please record the date of the first day of the course). 3.Which trainer(s) delivered the course? Tick all that apply. 4.How likely are you to recommend this learning event to a colleague with a similar learning need? a.Extremely likely b.Very likely c.Likely d.Somewhat likely e.Not at all likely 5.How would you rate the content covered in this session when considering the relevance to your role? a.Extremely relevant – I expect to use the lessons learned on a daily basis b.Very relevant – I expect to use the lessons learned at least weekly c.Relevant – I expect to use the lessons learned at least once a month d.Somewhat relevant – I expect to use the lessons learned but less than once a month e.Not at all relevant – I don’t anticipate using the lessons learned at all in my role 6.Which of the following statements best describes how you felt about the pace of the training? a.Much too fast b.A little fast c.About right d.A little slow e.Much too slow 7.Which of the following statements best describes how you felt about the amount of information provided? a.Far too much information b.A little too much information c.About right d.Not enough information e.Far too little information 8.How would you rate the style, interaction and structure of the course? a.Excellent – the style, interaction and structure fully met my learning needs b.Good – the style, interaction and structure mostly supported my learning needs c.OK – I have nothing positive or negative to say about the style, interaction and structure d.Poor – the style, interaction and/or structure hindered my learning 9.How would you rate the knowledge, support and approach of your trainer(s)? a.Excellent – the trainer(s) demonstrated a firm grasp of the topic, encouraged questions and adapted their delivery to suit the learners’ needs b.Good – the trainer(s) demonstrated a broad understanding of the topic, answered questions and regularly checked in with learners to make sure they understood c.OK – the trainer(s) demonstrated a narrow understanding of the topic, answered questions and checked in with learners to make sure they understood d.Poor – the trainer(s) demonstrated little or no knowledge of the topic and/or failed to answer questions and/or failed to check in with learners to make sure they understood 10.How would you rate your confidence in your level of understanding and ability to find additional support/information on this topic? a.Excellent – I can answer the most common questions about this topic and know where to find more detailed information if needed b.Good – I can answer most of the common questions about this topic and know where to find more detailed information when needed c.OK – I am not confident answering the common questions, but I know where to find supporting information to help me to do this d.Poor – I am not confident answering the common questions about this topic and don’t know where to find supporting information to help me 11.What do you think we did well in delivering this course? Let us know so that we can do it again! Remember – your response is anonymous unless you choose to share your email address with us below. 12.What would make our delivery of this course better? Remember – your response is anonymous unless you choose to share your email address with us below. 13.Occasionally it is useful for us to contact learners to obtain more detailed feedback. If you would be happy for us to get in touch for this purpose, please enter your email address here.
How is the data used?
​
-
Informally after each session: the trainer/facilitator will review the feedback to spot anything that may suggest that the session has not landed as intended
-
Monthly: all feedback for the previous month is reviewed and compared to that received over the previous year and/or lifetime of the course (whichever is longer) to identify trends in the feedback which may indicate areas that need improvement. A summary of the data is included in the monthly Academy reporting submitted to the management group.
-
Within the cyclical review process to help justify proposed changes to the content or structure of the course.
e-Learning feedback form
Form wording
Please take a couple of minutes to complete this feedback form based on your learning experience. We appreciate your openness and honesty - your feedback will help us to improve the content that we offer. By default, your response is anonymous. 1.Which course are you providing feedback for? (Please select the course from the list). 2.On what date did you complete the course? 3.How likely are you to recommend this learning event to a colleague with a similar learning need? a.Extremely likely b.Very likely c.Likely d.Somewhat likely e.Not at all likely 4.How would you rate the content covered in this session when considering the relevance to your role? a.Extremely relevant – I expect to use the lessons learned on a daily basis b.Very relevant – I expect to use the lessons learned at least weekly c.Relevant – I expect to use the lessons learned at least once a month d.Somewhat relevant – I expect to use the lessons learned but less than once a month e.Not at all relevant – I don’t anticipate using the lessons learned at all in my role 5.Which of the following statements best describes how you felt about the pace of the course? a.Much too fast b.A little fast c.About right d.A little slow e.Much too slow 6.Which of the following statements best describes how you felt about the amount of information provided? a.Far too much information b.A little too much information c.About right d.Not enough information e.Far too little information 7.How would you rate the style, interaction and structure of the course? a.Excellent – the style, interaction and structure fully met my learning needs b.Good – the style, interaction and structure mostly supported my learning needs c.OK – I have nothing positive or negative to say about the style, interaction and structure d.Poor – the style, interaction and/or structure hindered my learning 8.What was your experience of the course navigation? a.Excellent – I had no difficulty navigating through the course b.Good – I had no difficulty navigating the course, but still think it could be improved c.OK – Most of the course navigation was fine, but there was at least one point where I was initially unsure how to progress d.Poor – Course navigation was confusing and/or unclear 9.What was your experience of the course assessment(s)? a.Excellent – All of the content tested in the assessment was covered sufficiently in the course content b.Good – All of the content tested in the assessment was covered in the course, but not in sufficient depth c.OK – Most of the content tested in the assessment was covered in the course, but not in sufficient depth d.Poor – Content tested in the assessment was not covered in the course OR Questions were unclear or misleading e.This course did not include an assessment 10.How would you rate your confidence in your level of understanding and ability to find additional support/information on this topic? a.Excellent – I can answer the most common questions about this topic and know where to find more detailed information if needed b.Good – I can answer most of the common questions about this topic and know where to find more detailed information when needed c.OK – I am not confident answering the common questions, but I know where to find supporting information to help me to do this d.Poor – I am not confident answering the common questions about this topic and don’t know where to find supporting information to help me 11.What do you think we did well in delivering this course? Let us know so that we can do it again! Remember – your response is anonymous unless you choose to share your email address with us below. 12.What would make this course better? Where your comments relate to something outside of our control (such as the performance of Absorb or of external links), we will do our best to pass your feedback onto someone able to act on it. Remember – your response is anonymous unless you choose to share your email address with us below. 13.Occasionally it is useful for us to contact learners to obtain more detailed feedback. If you would be happy for us to get in touch for this purpose, please enter your email address here.
How is the data used?
​
-
Monthly: all feedback for the previous month is reviewed and compared to that received over the previous year and/or lifetime of the course (whichever is longer) to identify trends in the feedback which may indicate areas that need improvement. A summary of the data is included in the monthly Academy reporting submitted to the management group.
-
Within the cyclical review process to help justify proposed changes to the content or structure of the course.
Post-course evaluations
The opportunity to provide feedback about the course as a whole will be offered to the learners within the notification of course completion email (see above). Feedback collected at this stage is expected to be more holistic than that collected after each module, but the impact of the course on the learner will still be largely theoretical at this stage, as they have not had the opportunity to test their new skills in the real world. This feedback is expected to provide insight into how the overall course flows for the learner, rather than focusing on the individual modules that make it up. Feedback will be collected using the Academy e-Learning Feedback Form (see above).
3-month evaluation
Learner evaluation
Learners will be invited to meet with the lead trainer 3 months after their graduation to share their experiences of putting the theory into practice. The trainer will prompt learners to consider their experiences and reflect on how well they feel the course prepared them to deal with them. The meeting will be minuted by the lead trainer. Learners will also be invited to send in written testimonials for inclusion on the info page of the Academy’s Hub. The meeting will have an informal format to encourage discussion and honesty.
How is the data used?
​
Testimonials will be added to the Academy Hub’s dedicated info page (with explicit permission from the learner who submitted each one). Testimonials that are more than 18 months old will be removed and archived to ensure that potential delegates are seeing feedback from recent cohorts whenever they view the page. Testimonials may also (with explicit permission from the learner who submitted each one) be used as part of the ongoing course advertising through Viva Engage and the CEO's update.
​
Feedback will be recorded after each meeting and analysed to identify trends and potential areas for improvement to the learner experience. It will also be used within the cyclical review process.
Manager evaluation
Managers will be invited to provide feedback about the post-course performance of their staff 3 months after they graduate. Feedback will be collected using the Academy’s Training Effectiveness Review form.
Form wording
Please submit one response for each member of staff listed on the email. We will use this information to improve the learning interventions we supply. This form will take approximately 5 minutes to complete. Please answer the following questions in respect of the specified learning intervention only. 1.Which member of staff are you providing feedback for? (Copy and paste the name from the email) 2.Is this person still in role? 3.Which course did the team member attend? (Copy and paste the name from the email) 4.Please indicate how far you agree or disagree with the following statements about this team member and this learning intervention. a.The team member now has a good level of understanding of the subject(s) covered. b.The learning intervention has increased the team member's confidence in their ability to carry out their role. 5.Please provide details to explain your answer to the previous question. 6.Which of the following best describes the learner's relevant performance of the related role since their training? a.The learner's performance has exceeded expectations b.The learner's performance has met expectations c.The learner's performance has not met expectations d.The learner has not had an opportunity to prove competence in this. 7.What has prevented the learner from putting this learning into practice and/or meeting expectations? 8.Which individuals has the above named acted as a Buddy for since their graduation? 9.Which processes have they provided on-the-job training on as a Buddy? (Please enter the full name and code of each Pega work type trained) 10.Has the learning intervention added value to your area in the case of this learner? 11.Please provide details to explain your answer to the previous question. 12.Based on the learner's performance since they attended the session, was the training they received sufficient? 13.Is further support needed to embed this learning? 14.What further support is needed to embed this learning? 15.How likely are you to recommend this learning intervention to another team member or colleague? a.Extremely likely b.Very likely c.Likely d.Somewhat likely e.Not at all likely 16.If you have any other feedback about this learning intervention that has not already been captured by this form, please enter it here.
How is the data used?
​
This information will be used in conjunction with analysis of the identified KPIs (see part 1 and below) to judge the effectiveness of the course against the identified business need.
Business evaluations
The following reports will be commissioned from the Operational MI team to allow the effectiveness of the course to be tracked against the identified business need. See part 1 for information about the identified KPIs and an overview of how the WFM system's licencing works.
​
The ongoing reporting will allow the business to track the data to show what impact the roll out of the course has had on the performance. Information from the manager evaluation will allow us to identify which trainees were taught by a trained Buddy and to exclude data relating to staff not trained by a BEST Buddy Programme alumnus.
Report
Description
Benchmark
Frequency
Expected trend
Cases processed at 'Gaining Licence'
The report will show all staff who have been at a status of ‘Gaining Licence’ during the reporting period. For each work type that they were ‘Gaining Licence’ for during the period, the report will show how many items were processed and whether or not they were processed correctly. It will also show how many items have been consecutively processed correctly (longest run) and how many must be processed correctly consecutively for the trainee to gain their licence.
Data from the 12 months to <date to be agreed with project sponsor>
Monthly
We would expect the number of cases processed at ‘Gaining Licence’ to reduce towards the minimum required for each work type.
Reversions to 'Licence Under Review'
The report will show all staff who have made processing errors resulting in their licence for a work type being under review. For each instance, the report will show the user, the work type, the date they initially gained their licence, the date they were put under review, the number of cases processed while under review and the date that they regained their full licence. It will also show how many items have been consecutively processed correctly (longest run) and how many must be processed correctly consecutively for the user to gain their licence.
Data from the 12 months to <date to be agreed with project sponsor>
Monthly
We would expect the number of cases processed at ‘Licence Under Review’ to reduce towards the minimum required for each work type.
Reversions to 'Licence Under Review'
The report will show all staff who have lost their licence for a work type during the reporting period. For each instance, the report will show the user, the work type, the date they initially gained their licence, the date the licence was lost, how many items were processed at the ‘Licence Lost’ status, how many items have been consecutively processed correctly while at that status (longest run) and when the licence was regained. It will also show how many items must be consecutively processed correctly for the licence to be regained.
Data from the 12 months to <date to be agreed with project sponsor>
Monthly
We would expect the number of cases processed at ‘Licence Lost’ to reduce towards the minimum required for each work type.
NPS
(Net Promoter Score)
​
The report will show the rolling NPS for each work type for the 12 months to the reporting date.
​​
NB – NPS is affected by a wide range of factors and changes cannot be definitively attributed to the roll out of this course
Data from the 12 months to <date to be agreed with project sponsor>
Monthly
We would expect the roll out to have a delayed, but positive, impact on NPS.
Complaints
The report will show the date of the complaint, the concern type (levels 1 to 4), the product and the proposition for complaints closed with a root cause of ‘Care & attention’ or ‘Training’. Complaints that are still under investigation will be excluded.
​
NB – Complaints can take up to 8 weeks to investigate and resolve. Root cause data cannot be added to the incident until the investigation is complete. They are affected by a wide range of factors and changes cannot be definitively attributed to the roll out of this course
Data from the 12 months to <date to be agreed with project sponsor>
Monthly
We would expect complaints closed with a root cause of ‘Care & attention’ or ‘Training’ to go down for the work types that have been trained by a BEST Buddy.
In line with the implementation plan, these business figures will be subject to full analysis and reporting back to the business at the 3-month and 6-month mark after the course is piloted. After the 6-month report, the requirement will be reviewed to see whether this level of reporting is still required by the business (the data will still be monitored within the Academy for quality purposes.
Because of the nature of the data, further discussions with HR are required to agree what, if any, information could be provided in relation to staff retention, but the following data would be desirable:
​
-
Name (for on-going reporting, this would allow us to identify whether or not they had completed or applied for the BEST Buddies Programme, though this data would not be necessary for the benchmark)
-
Leaving date
-
Reason for leaving (this would allow us to estimate whether the provision of this course had any impact on the number of staff stating ‘lack of development opportunities’ as their reason for leaving).
-
Length of service (this would help us to estimate the impact of the course on the target audience – staff with at least 13 months’ service who are eligible to apply to progress to P&P zone 3)
-
Position in the P&P scheme (this would help us to estimate the impact of the course on the target audience – staff with at least 13 months’ service who are eligible to apply to progress to P&P zone 3)
Analysis of these reports would be fed back to the Customer Care management team to allow them to gauge the effectiveness of the course over time. In the first year, this data will be reported monthly. After that point, the management group’s reporting requirements will be reviewed.