On February 26, 2021, the Alberta Mentorship Program hosted our fourth bootcamp. Our Start-Up Bootcamps are a community learning space to cover important topics for new and growing mentorship programs. This session discussed how to evaluate the success of your mentorship program. Rosa Martinez, Mentoring Partnerships Coordinator at the Calgary Region Immigrant Employment Council (CRIEC) lead the bootcamp presentation supported by Bruce Randall, Executive Director of CRIEC and Doug Piquette, Executive Director of Edmonton Region Immigrant Employment Council (ERIEC).
Rosa has done a deep dive into program evaluation and shares what she learns with the CRIEC team. She has built their evaluation up from the basics to a comprehensive system.
“Rosa is constantly, 24/7 thinking about the work we do,” says Bruce. “She has inspired the whole team and shares with us. It is a vital role.”
Measuring Mentor and Mentee Benefits
Rosa worked with a researcher from Royal Roads University to evaluate CRIEC’s mentorship program outcomes. What it showed was that mentorship provides clear benefits for both mentors and mentees. These benefits included:
Professional Benefits
For both mentors and mentees, mentorship built confidence and career-related skills. Mentors increased their leadership and communication skills while mentees increased their job search skills.
“We measure mentees’ confidence by asking, ‘How many jobs do you think you are ready for?’” says Bruce. Before mentoring mentees will answer 20 jobs or even 100 jobs. After mentoring they will answer 5 jobs. This lower number demonstrates a better understanding of their skills and that their confidence is higher. Mentorship helps mentees to narrow and focus their job search.
The research also showed that mentees benefit most when they are ready for mentorship. If they are still working on language skills or focused on immediate needs, the information shared was not being absorbed and retained as well.
“It’s not a judgement. It just means a mentee might not be ready for mentorship until they hit that stable plateau,” says Bruce.
Read more: Recruiting Ready Mentees
Psychological Benefits
Mentorship increases professional relationship satisfaction and learning for both mentors and mentees. The connection they build through mentoring creates a bond and makes the participants feel good. They can also see how the mentoring relationship is different from a friendship and different from a relationship with a co-worker or manager. The intentional reflection and discussion about the mentee’s career path creates a relational learning environment that is unique to mentoring.
Mentees tell us, “I can now identify a potential mentor for myself going forward. I know what a good mentor sounds like, looks like, feels like,’” says Bruce. “This is reinforced in the research. We are equipping mentees to find their own mentors.”
Intercultural Understanding
Often, CRIEC begins working with mentees in the first days after they arrive. In the first moments, mentees do not yet understand the cultural differences in Canada, especially the professional differences. This understanding supports the mentees’ increased job search skills and their transition to the Canadian workforce.
Mentors also have increased intercultural understanding which better prepares them to work with internationally-educated professionals. When employees inside the company have increased intercultural understanding, this also smooths the transition for new internationally-educated employees.
Read more:
CRIEC’s Program Evaluations
Evaluations can compile a variety of information including:
- Quantitative data (attendance numbers, employment statistics, or demographics)
- Qualitative data (stories, testimonials, or experiences)
In combination, both types of data can provide a good picture of how well you are doing. Usually, your funder will be looking for quantitative data, but the qualitative data can be helpful to fill out the story of the numerical data. It can also be helpful to showcase your success stories to potential mentors and mentees.
CRIEC has developed different evaluation strategies for each of its programs.
For their core mentoring and networking programs, CRIEC does a range of evaluations from surveys, interviews, and focus groups. This information helps the CRIEC team to respond to participants' needs and to formally capture learnings from each session in a way that can be shared with the whole team.
CRIEC’s Smart Connections is like speed dating but for mentoring. Attendees get short sessions with a variety of people. The goal of the event is to provide experiential learning and create engaging networking. While it does not have the depth of the long-term mentoring relationship, it can illuminate gaps for mentees – especially if a mentee received the same feedback several times. To evaluate the success of these events, CRIEC uses scorecards and session surveys.
“We tend to stay away from asking 'was the room good' and 'did you like the food',” says Rosa. "Instead we ask, 'What was your main takeaway?'”
The responses can help to shape future educational events. For example, if we see “my stories are too long” from a lot of mentees, CRIEC can offer a session on storytelling helping mentees to tell stories in a Canadian professional context.
These scorecards can also collect data and demographics that can be used for reporting to funders.
Meeting Funder Requirements
Ideally, evaluation is a reflective process that guides your organization to grow and improve, but meeting funder requirements is likely a critical part of your evaluation.
“We don’t love counting the numbers, but we need to give the information to the funder,” says Bruce.
Whenever funding is provided, there are targets your organization will be expected to meet. These could include:
- Activity Targets (holding a specific number of events or connecting a specific number of matches)
- Target Outcomes (meeting specific benchmarks)
Each funder will have different outcomes and different expectations for the evaluations and targets that you will need to achieve and that are based on the funding organization’s goals.
For example, CRIEC has clear targets that they must meet for their funding from Immigration, Refugees and Citizenship Canada (IRCC). During 2020, they were expected to complete the following activity targets:
Mentorship:
- 130 mentoring matches
Networking Opportunities:
- 130 Smart Connections
- 90 coffee meetings
CRIEC was also expected to achieve the following target outcomes:
- 65% of clients have secure employment or are following an appropriate educational path three to six months after mentoring
2020 was a different year with the pandemic, but CRIEC was able to shift to online programming and mentorship. They were able to meet their targeted outcomes despite the challenges of last year.
Over time IRCC has become more explicit about wanting quantitative data (the numbers). CRIEC did not always formally track numbers. But as they did more formal tracking and reviewed past years, they were able to see that they have been achieving that 65% target outcome for some time. Even in 2020, “we are exceeding what the funder wants,” says Rosa.
“Mentoring is not just a job thing. It’s not a short-term fix. Mentoring is a long-term fix. You want to create a society where you have unbelievable talent working where you want it to be working,” says Bruce.
Doing evaluation requires finding a balance. Doing what you need to do to meet funder requirements, but also doing what you need to do to meet your mission. It can also help to focus your team to work together to meet the same targets and goals.
“It is all about balancing IRCC outcomes, our outcomes, and the mentee and mentor needs,” says Rosa.
Creating Your Own Evaluation Plan
An evaluation plan can be a careful and comprehensive strategy. However, when you are starting out with your program, you can start simple and build it as you grow. Both CRIEC and ERIEC have taken 12 years to create more formal evaluation plans. This is a lot of years of learning, tweaking, and growing the evaluation. It is not where they started.
“This is a language we are used to after 12 years of working on it,” says Doug. “For new folks, this might be really daunting. In time, you will get there and understand these challenges.”
There are basic strategies around pre- and post-survey forms,” says Cheryl Whitelaw, Project Manager of the Alberta Mentorship Program. “Ask broad questions like ‘Do I feel more connected to this community?’ or ‘Do I feel like I know where to network?’ To help you get started, you can find our free evaluation surveys and tools to use as-is or adapt to your organizations’ mentorship program (see links below).
“Where we are now, is a good place to be, but we’ve been open to learning continually,” says Rosa. You will find that as you have closed one gap or corrected a problem, that there will be new gaps or improvements exposed. “We learned that gaps in the services and needs could shift," says Rosa. What worked yesterday might not work today or next week as the economy shifts or the types of mentees coming to CRIEC changes. She sees evaluation as a constant process of doing, reflecting, learning, adjusting your plan, and doing again.
The CRIEC team has really worked on following this cycle together. If you start with basic surveys and collect data about the number of mentors and mentees and where they are at in 6 months, you will begin to accumulate some data. Take the time to collect and reflect on what the data shows. Consider the survey comments or informal feedback from your mentors and mentees. This reflection can show you what you are doing well and what can be improved.
In preparing this presentation, Rosa took some time to reflect on how CRIEC’s evaluation and the program have changed over time. “It was interesting to go back and look at it again,” she says. “It reminds you of all the work you have done and are doing.” While it is good to look at what you can improve or change through feedback, it is just as important to reflect on what you are doing well and how many people you are helping.
Feedback and evaluation are learning tools that can help your team grow and improve. No matter how long or established your program is, it will always require adjustments. Sometimes that is because there are better or different ways to approach the program. Sometimes it is because the needs of the mentees and mentors change with different societal, economic, or political shifts.
“Often the funder is most interested in the shorter term and whether the mentees getting jobs related to their education,” says Bruce. “Good mentoring programs focus on the immediate and intermediate outcomes which will help this job but also the next job.”
Mentorship is about setting up those skills that will help mentees in the long term. We want to see mentees get hired and transition into a job as soon as possible, but we also want them to be able to use those skills to develop their career within a company or for their next job. Mentorship can set a mentee up for long-term success.
As part of that, CRIEC evaluates mentoring success with several types of outcomes:
- Aligning mission & mentoring delivery
- Being intentional in implementing effective mentoring activities
- Fostering team learning to ensure outcomes are the goal of mentoring
- Iterating and learning from initiatives
- Striving for an organizational culture that embraces diversity
- Promoting lifelong learning as the key to deal with change
"You are in a different community, so what you need to evaluate might be different,” says Rosa. Finding the right outcomes to measure and the right ways to evaluate them will take some trial and error. “Believe in yourself. Be iterative. Learn and share as a team. Learn from other organizations,” advises Rosa.
CRIEC and ERIEC have shared their learnings about evaluation with each other and both organizations are ready to share what they have learned and to provide resources and tools that they have developed with other Alberta mentorship programs.
“Don’t forget to collect the stories,” says Rosa. “One-on-one interviews are great.” It takes time but builds the success stories of your program that are inspiring to your team and to potential mentors, mentees, and partners. These stories combined with data you collect from surveys can create a well-rounded picture of your program.
Getting started with evaluations can be as simple as starting with some pre- and post-program surveys and collecting participant stories. Start with the basics and build up an evaluation strategy as you grow and gain experience.
Read more:
You can find stories from Alberta Mentorship Program’s pilot partners on our YouTube channel.
Download our Evaluation Tools:
We encourage you to use any forms we provide as they are or to customize them for your program needs, including adding your logo. If you would like to customize these forms, you can find the full set of editable forms here.
Developed with support from the Government of Alberta - Labour and Immigration, Workforce Strategies.