Comprehensive Reflection

How has EDTEC affected me in my professional life as an educator? Interestingly, I wouldn’t have called myself an educator five years ago and that is perhaps the largest single effect of my time in the program. I can speak the language of education, and can read and understand educational research. I no longer feel like I am blindly throwing a dart at a moving target when I develop instruction. For years, my work as a librarian presented opportunities to educate patrons about various information literacy skills, but I felt woefully unprepared to design and develop effective instruction. Now my toolkit is filled with fabulous theories, practices, and techniques, along with a systematic approach to the entire process. Three of these have been particularly enlightening and invigorating as I’ve developed myself as an educator:

• Performance analysis
• Evidence-based Practice
• Evaluation

In the next few sections I’ll reflect on the importance of these tools and how they inform my educational technology practice.

PERFORMANCE ANALYSIS

Performance analysis was new to me when I learned about it in EDTEC 540, and I couldn’t believe that it wasn’t more widely known. In the eleven years between obtaining my first graduate degree and starting my second, I’d worked in three industries (public utility, nuclear engineering, and healthcare) in a variety of positions. Every single place I worked, managers and HR professionals automatically assumed that a performance problem needed a training solution. The idea that training might not be the appropriate response never even occurred to most decision makers.  After reading Mager’s Analyzing Performance Problems in the summer of 2004, I knew that I wanted to do a “real” performance analysis project within my organization.

I’d recently been named co-chair of our hospital patient education committee, and the work of the committee was ripe with opportunity for performance analysis and improvement. I took on an issue that had been plaguing our inpatient units for a long time – education for newly diagnosed diabetics. Since the elimination of certified diabetes educators (CDE) a few years prior, staff nurses had been responsible for educating patients who had just been diagnosed with Type 1 or Type 2 diabetes. But the staff nurses weren’t doing it, and no one really knew why. Too little time? Not enough up-to-date knowledge about diabetes? Lack of teaching materials? My performance analysis was the first attempt to understand the problem and create relevant solutions; up until then, the nursing education department just kept saying, “We need to create an in-service about that.” It turned out that only two of eight factors were related to a skills or knowledge gap – the other factors were related to motivation, logistics, or organizational support. Education would not have solved the problem.

This opened my eyes (and a lot of other eyes as well) to the myriad of performance improvement opportunities that exist with only a little bit of targeted, up-front analysis. I eagerly read Allison Rossett’s First Things Fast and learned about ways to do “speedy” performance analysis. From that first project, I started practicing my analysis skills whenever I had the chance. Residents not attending Grand Rounds as expected? Let’s do a speedy analysis. People aren’t using the monthly reports I send out? Let’s do some quick telephone interviews and find out why. I wouldn’t go so far as to say that I became a performance consultant for my organization, but I did promote the use of up-front analysis any chance I got.

Three years ago I changed jobs and am no longer in a position to evaluate or analyze very many performance issues, because I am a telecommuter who sees only a tiny piece of each product that we produce. But performance analysis still remains my largest and perhaps most lasting take-away from the EDTEC program. I continue to seek opportunities for performing up-front analysis outside of my work environment.

For instance, I think there is a role for performance analysis when designing instruction for teaching a brand new skill – even before a performance gap has the chance to occur. Reviewing the literature and reading case studies of similar initiatives may reveal common performance problems, problems worth addressing when designing similar programs. Over the last few years I’ve worked with colleagues in my professional association to propose new CE course ideas. As we wrestle with learning objectives and pinpoint what we want learners to demonstrate at the conclusion of each workshop, I like to highlight data (if it exists) about common performance issues surrounding the particular skill or knowledge base we are creating. Knowing about performance gaps can help instructional designers mediate some of them from within the instructional strategies themselves.

This focus on literature and existing research data brings me to the second of my key EDTEC themes, evidence-based practice.

EVIDENCE-BASED PRACTICE

The next aspect of educational technology that has profoundly affected how I think about and practice instructional design is evidence-based practice. Once performance analysis reveals that training is a viable solution for a performance problem, how does a designer proceed? If you’ve ever been a student (and that should cover 100% of potential readers of this reflection), you’ve experienced education that “works” and education that fails miserably. Why do some initiatives work and others fail? Determining the answer to that question was one of the reasons I enrolled in EDTEC classes and has intrigued me from the beginning of my journey.

I was excited to see that the field of education  in general – and educational technology in particular – is starting to do evidence-based practice (EBP) and make design decisions based on pedagogical research evidence and not just intuition or expert opinion. I come from a healthcare background, where evidence-based practice has had a foothold since the early 1990s. While research in education tends to incorporate more qualitative analysis than does clinical medicine, many of the principles of EBP as practiced in medicine do apply to educational interventions.

An evidence base seems more readily available in K-12 education than in other educational sectors, which is moderately frustrating but understandable. For instance, Johns Hopkins University established a Center for Data-Driven Reform in Education and publishes systematic reviews in its Best Evidence Encyclopedia (http://www.bestevidence.org).

Fortunately, the last decade has seen a push toward evidence-based practice in the design and development of e-learning. Ruth Clark is a major advocate for using research evidence to guide the design of learning interventions. Three Clark texts on my bookshelf take an evidence-based approach: The New Virtual Classroom (Clark & Kwinn, 2007); Graphics for Learning (Clark & Lyons, 2004); and e-Learning and the Science of Instruction (Clark & Mayer, 2003). Clark is a leader in EBP and has published others with an evidence-based focus. I was so pleased to learn about her work through various classes in the EDTEC program.

Others are jumping on the EBP bandwagon. In September 2010, the U. S. Department of Education released a meta-review entitled Evaluation of Evidence-Based Practices in Online Learning  (this 2010 report corrects a 2009 version with multiple errors). The investigators reviewed more than 1,000 studies published between 1996 and 2008 and identified more than 50 “effects” eligible for meta-analysis. The study’s findings were many, but particularly relevant to my interest in selecting research-proven interventions was the finding that of 13 different online learning practices, only two had a measurable positive effect on learning. Blending online with face-to-face instruction was found more effective than online learning alone, and the practice of instructor-led rather than self-directed learning was significantly more effective. Other practices had no significant effect, either positive or negative. Video? No effect. Online quizzing? No effect. A few practices were found to have a negative effect. Since my work as an online adjunct often includes overseeing group projects, of great interest was the finding that learning guidance offered to online groups appeared less effective than the same guidance offered to individuals.

What this U.S. DOE report says to me is that practicing evidence-based education leaves plenty of room for experimentation and innovation within certain research-defined parameters. This meta-analysis does notindicate that we shouldn’t make video or online quizzes part of our course designs. What it says is that whether or not we use them should not turn into a key decision point in our designs. And just because group learning isn’t as effective as individual when it comes to content mastery, it does not mean we should not plan for group projects. There are multiple other benefits to group work, such as learning to manage social interactions, coordinate projects, or reach consensus.

I have no desire to reinvent the wheel. If research has already shown that a particular practice is more effective than another, I’m going to select the first practice in my instructional design. Secondly, I realize my own limitations regarding the ability to perform good quality research – or even to do critical appraisal of existing research. If someone has already examined and summarized the research evidence on a particular educational intervention, I am more than willing to utilize their knowledge than to attempt to build that knowledge from scratch. In these days of ever-shrinking budgets, educators must be sure that selected instructional methods and practices are effective.

In clinical medicine, just because a treatment is demonstrated effective for most patients doesn’t mean that it will work for a particular patient.  Educational interventions proven effective in certain settings or with certain types of learners may not translate well to your setting or audience. Which is why it’s crucial to consider the third of my key EDTEC themes…evaluation.

EVALUATION

About the time I started the EDTEC program, I learned about evaluation within my discipline of medical librarianship. Catherine Burroughs, who was then affiliated with the Pacific Northwest office of the National Network of Libraries of Medicine, wrote and published a free guide called Measuring the Difference which focused on planning and evaluating health information outreach projects. At that stage in my career, I was only a little bit familiar with what I later came to know was summative evaluation and also the use of Likert scales at the end of an educational intervention to gauge whether or not each participant enjoyed the experience and thought they learned something. Evaluation, like performance analysis, was mostly a new concept to me and I was intrigued (and also a little confused) about how to plan for evaluation in the earliest stages of a project.  

Not long after I read Measuring the Difference I learned about Kirkpatrick’s levels of evaluation in my EDTEC courses. I understood instinctively that measuring learning transfer took more time and resources than doing Level 1 or Level 2 evaluations. But which would be more expensive in the long run: performing substantive evaluation, or dealing with an audience of students or workers who did not have the necessary skills to succeed in their studies or in the workplace? I embarked on a journey to learn as much about evaluation as I could, including the knowledge of when certain methods and levels were appropriate and when they were not. I started with Donald McCain’s book Evaluation Basics, which offered a fantastic introduction to the concepts and processes surrounding evaluation. [As an aside, the indexer in me cringes that the ASTD Training Basics Series books all lack indexes, significantly reducing their usability] I learned about the difference between formative and summative evaluation in EDTEC 544, and was delighted that I was able to incorporate formative evaluation into an intranet redesign project at work.

Coinciding with my interest in evaluation were two adjunct appointments. In 2005 I was appointed to teach graduate information literacy at the University of Maryland University College, and in 2007 I joined the adjunct faculty at Drexel University in order to teach health science librarianship. Higher education presents a complex evaluation picture because unlike workplace learners, you cannot test students after the close of a semester in order to evaluate the far-transfer of the taught skills or the broader impact such as encouraged by a Kirkpatrick Level 4 evaluation. This is a conundrum with which I still wrestle, and I’m not sure of the solution. Some of my colleagues argue that we are not teaching “skills” as much as we are teaching how to learn and an approach to knowledge acquisition. But even then, how do we know that we’ve successfully taught a student how to learn?

My career goals include learning as much as I can about evaluation. It is a practice with tremendous potential to impact my instructional design approach (I know only enough right now to make me dangerous). In EDTEC 590 we learned about program evaluation, an area that interests me because I have a gut feeling that evaluating programs in a broadly based, holistic manner will offer the greatest potential to impact education.

THE FUTURE

What does the future hold for performance analysis, evidence-based practice, and evaluation? It's hard to imagine that they will need to be retired as they become "old and stiff" as our guidance for this reflection suggests that we consider. I think that perhaps the largest change will occur in evidence-based practice. Currently, EBP utilizes quantitative evidence more fully than qualitative, because of the difficulty in doing meta-analysis on qualitative data. Practitioners will likely discover new ways of combining and analyzing qualitative data from multiple studies in meaningful ways. I also think that the bank of education-related questions that EBP seeks to answer will grow in higher education, workplace learning, and in other settings beyond K-12.

Evaluation has been in the spotlight this last decade as dissatisfaction with No Child Left Behind (NCLB) grows. Policy makers and regulators are starting to understand - as educators seem to have done from its institution - that the state-mandated assessments required by NCLB offer a one-size approach to a many-sizes problem. The focus on standardized tests and ability to manipulate outcomes by the way one classifies certain data points is also problematic. In the future, I think educators, policy makers, and regulators will come together to creatively develop new ways of assessing learning outcomes that offer greater flexibility in both what is assessed and how we define success.

The fantastic thing about Educational Technology is that it is so broad as to affect learning in all settings and for all people. The frustrating thing about Educational Technology is that it is so broad as to affect learning in all settings and for all people -- making it very difficult to keep up with new research, policies, and technologies. But it is a challenge I will relish as I move on to the next stage in my development as an educator.

REFERENCES

Burroughs, C. M., & Wood, F. B. (2000). Measuring the difference: Guide to planning and evaluating health information outreach. Seattle, WA: National Network of Libraries of Medicine, Pacific Northwest Region. Retrieved from http://nnlm.gov/evaluation/guides.html#A1

Clark, R. C., & Kwinn, A. (2007). The new virtual classroom: Evidence-based guidelines for synchronous e-learning. San Francisco, CA: Pfeiffer.

Clark, R. C., & Lyons, C. (2004). Graphics for learning: Proven guidelines for planning, designing, and evaluating visuals in training materials. San Francisco, CA: Pfeiffer.

Clark, R. C., & Mayer, R. E. (2003). E-Learning and the science of instruction: proven guidelines for consumers and designers of multimedia learning. San Francisco, CA: Pfeiffer.

Mager, R. F., & Pipe, P. (1997). Analyzing performance problems: Or, you really oughta wanna (3rd ed.). Atlanta, GA: Center for Effective Performance.

McCain, D. V. (2005). Evaluation basics. Alexandria, VA: ASTD Press.

Rossett, A. (2009). First things fast: A handbook for performance analysis (2nd ed.). San Francisco, CA: Pfeiffer.

U. S. Department of Education, Office of Planning, Evaluation, and Policy Development. (2010).  Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Author. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf