Today’s chief learning officers need to have access to meaningful information for decision-making purposes. This data should be appropriately organized around a common set of indicators that provide a broad perspective on all areas CLOs manage.

CLOs should have four main quadrants on their learning analytics dashboards: operational, financial, performance and cultural. Achieving success with learning analytics dashboards also requires five primary steps: research learning metrics, identify macro learning constructs, build micro learning indicators, build a process to collect and report, and design technology and templates for support.

Research Learning Metrics
The first step is to learn what others have done, so you don’t reinvent the wheel. As a CLO, an important best practice is to instruct your team members to do their homework before creating highly custom, internal dashboards.

Identify Macro Learning Constructs
A macro learning construct is a small, well-balanced set of broad learning metric classifications that summarize the results of the entire L&D organization. The four macro learning constructs are operational, financial, performance and cultural.

Operational constructs determine how much individuals trained. It is mostly volume or activity data, typically in a learning management or registration system.

Financial constructs answer the question, “What’s the benefit, cost or income statement impact?”

Performance constructs determine how well individuals were trained. They look at results-oriented metrics.

Cultural constructs answer the question, “How conducive is our environment to training that is perceived as organizationally strategic?” This is important to stabilize the learning and development organization during major change.

Study the macro learning constructs and ensure they fit with your organization. It is better to have fewer than more, so if some don’t work for you, cut them. Just ensure you maintain balance.

Build Micro Learning Indicators
Micro learning indicators are quantifiable performance measures that are tracked and linked to the macro learning construct. The micro learning indicators are the actual key performance measures for the four learning constructs.

Operational indicators include:

Number of students trained.

Instructor use rate.

E-learning use rate.

Average class size.

Staff-to-management ratio.

Delivery mix.

Survey response rates.

Class completion or cancellation rates.

Sources for these data include the learning management system or registration systems. Another great metric here is e-learning use rate. This is the percent of the e-learning library actively used. Research shows it is less than one-third. As such, a lot of waste exists, and opportunities for streamlining are available.

Financial indicators include:

Cost per student day.

Learning and development cost as a percentage of payroll.

Learning and development budget to actual budget.

Learning and development investment mix.

Revenue growth.

Human capital contribution margin.

Productivity.

Learning and development ROI.

Some measures traditionally are tracked, such as cost per student day and learning and development budget to actual budget. CLOs, however, also should track metrics such as revenue growth, human capital contribution margin (payroll + learning and development expense / revenue), productivity (revenue / number of employees) and learning and development ROI.

Out of these, the most import is productivity, at least to CEOs in how they view learning and development. This is a financial metric that shows how much top-line output (revenue) can be generated per person. The more that number trends positively, the more productive human capital must be. A learning and development dashboard must have this number because improving human capital performance and productivity is the ultimate financial measure of that.

Performance indicators include:

Level 1 satisfaction scores, including instructor performance and courseware quality.

Learning effectiveness (test scores).

Time-to-job impact.

Business results linkage (sales, quality, cycle time, productivity, customer satisfaction, employee turnover, cost savings and risk mitigation.)

ROI (if not in financial construct).

Performance metrics determine how well an organization trains, so even the basic, Level 1 smile sheet data points are included here. A key metric is time-to-job impact. This is a post-training performance metric, and it is gathered on the job. It looks at what percent of people applied the learning and when. Research shows 55 percent of learning is not optimally applied within six weeks. This is wasted training, and removing it is a way to use the metric to rid your organization of scrap and save hundreds of thousands of dollars.

The business results listed here can be tracked in two ways. First, you can do a study to isolate the training impact to the result for a strategic, visible or costly program. Second, just track the trends in business results that are important to the larger organization and/or have strategic learning assisting in their outcome.

Cultural indicators include:

Training eligibility.

Average hours of training per employee.

Available tuition reimbursement.

Management support.

Internal validation.

External validation.

There are traditional metrics such as the percent of personnel eligible for training or hours per employee. But the last three are important too. Management support is collected on manager or participant follow-ups. It looks at how conducive a participant’s environment is before and after training. Internal validation is the number of internal case studies that articulate the success of a major program (try to do one to three per quarter). External validation consists of the articles, awards, speaking events and panels on which learning and development members serve. This shows the internal senior management learning and development is valued by industry peers and also should be valued internally as a strategic partner.

As a CLO, make sure your organization can provide these metrics in a reliable, consistent and timely manner.

Build a Processto Collect and Report
The success factors in this step include:

Start with available data.

Verify data are credible.

Create templates to store and track.

Conduct routine analysis (monthly).

Report key findings (quarterly).

Begin with data that are credible and easy to gather. Then build a standard template for the macro and micro variables to reduce rework and increase consistency and comparability. The micro indicators might be tracked monthly, but consider reporting them quarterly to senior management, preferably as a trend against actual results and against goals with some type of coding for analytic interpretation.

As a CLO, ask for the process or plan of action to collect and report the data. Ensure they will meet your needs for timely data within the budget parameters you’ve allocated for this type of work.

Design Technology and Templates for Support
The final step is to automate the collection, storage, processing and reporting of what you’ve built. Learning analytics technologies are great tools to build dashboards, input or import results, and track and trend with interpretive color-coding for analysis.

As a CLO, study automation and technology options after you’ve seen the information and used it awhile (at least two quarters). Automate the process when you are confident it is stable and functioning as designed.

You also should review your metrics. Ask yourself, “Am I getting the data I need to manage my business and show value to stakeholders, and am I getting these data in a timely manner?”

When answering this question, remember the operational, financial, technology and cultural metrics discussed. Do these exist in harmony with one another? Are some being missed?

These questions, and their answers, are important. You can only manage what you measure, so measure the right stuff.

Jeffrey Berk is the vice president of products and strategy for KnowledgeAdvisors, a learning analytics technology company that helps organizations measure, communicate and improve the effectiveness of learning and human capital investments through technology and consulting solutions. He can be reached at editor@clomedia.com.

According to the U.S. Census, American workers now spend more than 100 hours a year commuting. Not only is this “new” workforce more mobile, it is more decentralized — a single manager might have employees in several areas of the country, or even scattered around the globe, all of whom need to receive the same information at the same time.

These new workers are far more technologically savvy, as well. Totally accustomed to computers, cell phones, personal electronic devices, mobile audio players and the Internet, they expect instant communication.

They also multitask as a matter of course. They see nothing unusual about sending and receiving instant messages, making phone calls, listening to music, reading and replying to e-mail and writing reports all at the same time over a latte at Starbucks.

Always busy, they carry their work with them and consider downtime (such as the time spent waiting for an airplane or an appointment) as an opportunity to get something done. Yet, even though they put in long hours, they never seem to have enough time. As a result, many of the traditional approaches to corporate education don’t meet their needs. It’s hard for them to see the value of time spent sitting in a workshop or seminar. Even much of the e-learning organizations spend a great deal of money to produce or purchase is too slow and plodding for their fast-paced, fluid environment. They want — and need — information delivered as quickly and efficiently as possible.

Starting Young
Richard Sweeny, university librarian at New Jersey Institute of Technology, has done research on the newest generation of college students, whom he refers to as “Millennials.” His studies of those students born between 1979 and 1994 show that:

This generation is the most racially and ethnically diverse generation in U.S. history.

Thirty percent of the population is considered auditory learners (versus 65 percent who are visual and 5 percent who are tactile).

They want options and customization in every aspect of their lives.

They hate to waste time and want to learn quickly. They rarely read — or need — instructions and prefer to learn by doing and interacting.

They want to be mobile.

They were raised on computers and adapt faster to new technologies than any previous generation.

The challenge for corporate learning organizations is threefold: How to deliver a learning experience when and where these new learners need it, how to deliver it in a format that works for them and how to ensure the content is kept up to date.

Podcasts rapidly are becoming a viable option. They use the electronic media that are integral to the 21st-century environment. They present information in the snippets that members of the new workforce prefer. And they offer an alternative for addressing many of the realities of today’s learning landscape.

Leading the Way
Some trailblazing organizations already have begun to take advantage of this new form of media to improve the performance of their workforce.

IBM uses podcasting internally to deliver information to its employees and is now publishing real-time updates for its investors, so they can keep up with IBM’s take on the future and direction of business and IT. The new series, “IBM and the Future of ...” began with a podcast on IBM and the future of driving (http://www.ibm.com/investor/viewpoint/podcast/05-08-05-1.phtml).

The “Negotiating Tip of the Week” podcast series, featuring Dr. Josh Weiss from the Program on Negotiation at Harvard, has been downloaded more than 280,000 times since April 2005.

Herbalife, maker of nutritional supplements, is creating podcast training programs for its distributors around the globe. The company already has given away more than 1 million iPods to its employees.

The Otter Group recently started offering custom podcasts for clients. The venture has proved extremely successful. Otter Group founder Kathleen Gilroy said earnings from custom podcasts already total more than $100,000, a significant return on her initial $15,000 investment in the initiative. The Otter Group’s latest venture, “Everybody’s a CEO,” helps businesspeople go “Back to School 2.0” and learn the importance of — and how to implement — new media, including blogs, RSS feeds and podcasts.

In 2006, General Motors launched FastLane radio as an offshoot of its FastLane blog. This radio-style podcast — featuring interviews with top company executives in design, engineering and marketing — offers an inside scoop on the largest auto company in the world (http://fastlane.gmblogs.com/).

Drexel University in Philadelphia produces a podcast called the “Drexel e-Learning Minute.” It helps students who are new to online education address various issues and provides the prospective online learner with tools for success.

Financial services provider Capital One hands out iPods as standard equipment for employees who are enrolled in targeted training programs and makes podcasts available on the company’s intranet and corporate Web site. The driver behind Capital One’s use of podcasts is a vexing deficit, not in dollars and cents but in time.

Pal’s Sudden Service restaurants, based in Kingsport, Tenn., use audio and video podcasts to train all their employees. “Podcasts give us the capability to better train today’s generation of employees,” CEO Thom Crosby said. “They can watch or listen to a 30- or 60-second podcast right at their work station.”

Plugging into Learning
The adoption and practical application of podcasting as a learning option seems to be inevitable. As the technology becomes more commonplace in the larger environment, it seems natural the corporate learning function will follow. It’s been estimated that in 2006, 700,000 households in the United States used podcasting. That number is projected to grow to 12.3 million households by 2010.

To put this into context, 11 million households in the United States have adopted the MP3 format. The expectation is the number will grow to 34.5 million households by 2010. Research shows people are adopting podcasting as a listening option faster than they adopted the use of MP3 players. So, it’s not a big stretch to assume that about a third of the owners of MP3 players will be listening to podcasts in four years — not merely plugged into music as they commute, garden, exercise or work.

It’s important to keep in mind, of course, that podcasting cannot replace all the other types of learning-delivery methods an organization offers. As with a radio program or an audiotape, podcasting provides information in a one-way format. Because podcasting does not allow for interactivity and feedback, it is more like a lecture or an explanation than a stand-alone learning program. It’s best suited to provide just-in-time information about topics that are subject to frequent change and to expand and reinforce what people learn in seminars, workshops, self-study and e-learning programs.

But when used with other training or performance improvement methods, podcasting can be a very useful tool to achieve a variety of workforce learning objectives:

Reinforce training. Podcasts are excellent ways to help people retain information they have learned through other methods — a significant factor for the 30 percent of the population who are auditory learners. For example, podcasts that provide summaries of key learning points or lectures allow people to review the material as often as needed.

Supplement training. Not everything can be covered in a workshop or seminar. Podcasts can offer supplementary lectures, interviews or case studies that build on what people learn in a classroom or other training program. Podcasts also can be used as substitutes for between-class reading assignments.

Follow-up training. Once people have participated in a training program on a specific topic, podcasts can be used to provide additional knowledge. For example, podcasts can deliver updated content on a previously taught topic and suggest ways the learner can continue to apply the learning.

Provide information to people who cannot attend a training session. Podcasts can provide the text of lecture materials, highlights of key points and other content for people who need the information provided in a workshop or seminar but are unable to attend.

Test preparation. In many industries, people need to take tests or certification examinations. Podcasts can help them prepare by providing a convenient way to review the information they have learned in workshops, seminars and self-study programs.

Replace content-only portions of training programs. Classroom-based training, online training programs and self-study materials all include portions that simply deliver information. That information can be delivered via podcast, freeing up valuable training time for activities that require interaction, practice and feedback.

Podcasting has virtually limitless potential as a tool to improve workforce performance and enhance the effectiveness of learning and development initiatives. To make the best use of this tool, however, learning organizations need to understand what this dynamic medium can accomplish, what its limitations are and how to integrate them with more traditional forms of enterprise education.

Kaliym A. Islam is the director of development and technology services for the Depository Trust & Clearing Corp. He can be reached at editor@clomedia.com.

What is Podcasting?
As with any new high-tech buzzword, “podcast” has a certain amount of mystique associated with it. But the reality is quite simple: It’s basically an audio file. “The New Oxford American Dictionary” defines “podcast” as “a digital recording of a radio broadcast or similar program, made available on the Internet for downloading to a personal audio player.” According to Wikipedia, the term (a combination of “iPod” and “broadcast”) was first noticed by a mass audience when Ben Hammersley used it in an article in the Guardian on Feb. 12, 2004. As with “radio,” “podcast” can refer to both the content and the method of delivery.

A podcast, then, is a digital audio program. It’s a multimedia computer file that can be downloaded to a computer, an iPod or other compatible device and then played or replayed on demand. Updated content and new “editions” can be downloaded quickly and, in some cases, automatically. Given the functionality of the new iPod models, there’s room to expand the definition to include both audio and video content.

“Know me, know my business” is a refrain that should be known, absorbed and emblazoned in the brains and actions of every CLO. This appeal applies equally to the corporate business being served and to the business of the learning function.

A challenge for new CLOs is to identify the learning function’s business in the context within their new companies. The first step is to ask a set of key questions. How well these questions are answered and integrated will significantly affect the CLO’s decision-making process, effectiveness and path to success.

Analyzing the development services of the learning function involves 40 percent description and 60 percent integration. Description is used to create useful pieces of the overall picture. Integration is necessary to produce an understanding of how the function really works.

The integration process is interpretative and much like the interpretation of an old master’s painting, it requires a good knowledge of the context and subtleties of the particular function under review. This integration is incumbent on the CLO’s knowledge of the company and the training organization, and it can improve over time.

So, description dominates initially. The key questions that need to be asked and answered in this part of the analysis process are interrelated and reflect the interconnected nature of the overall environment of the learning function. A new CLO can begin the analysis at almost any point and will be led naturally to neighboring elements of this complex environment.

The Key Description Questions
How Does the Training Function Define its Development Clients? The CLO has at least two sets of clients to address when planning the function’s products and services. The first set consists of development clients, who are the executives who authorize and pay for program development. The second set consists of training audiences, who are the end-users and buyers of the training, job aids, coaching and other services and products offered. Organizations relate differently to their clients, and it is important new CLOs and their staffs have a clear understanding of their clients’ needs, wants and buyer values. Research on the latter can help find these answers.

In large, diverse companies, it’s possible to narrowly target the development clients. Through studying the company’s plans and the industry’s environment, the CLO might be able to position services to support an emerging need in the organization. It’s important to describe the target clients demographically: Who are they, and what roles do they play? In terms of integration, it is also important to describe them from a psychographic perspective: What is it about the way they think and what they care about that should affect the service, product, operating and delivery decisions to be made?

What are the Core Service Offerings and the Norms for Delivering Them? With this question, the issue is not the definition of specific courses or services offered (these, though important, are obvious and easily detailed) but defining the underlying concepts of the products and services and the norms that are established for the interactions between training staff members and with their clients. The CLO needs to align the function’s service strategies and norms with the culture of the company. The strategic and tactical nature of the services should be described and identified.

The CLO should ask, “What distinguishes my training function in terms of the services we provide and how we go about it? Are the training developers essentially being responsive to requests from key management, or are they engaged in some different way, perhaps consulting with management on curricular development or participating in corporate strategy development and long-range planning and interpreting that information to establish training needs and direction? What do the staff members define as good service? What do our clients define as good service? Do we have clear standards for performance that are either formally or informally imposed?”

Answering these kinds of questions will help to draw a picture of the service concept and norms of importance. And such description helps motivate staff members to do the right things and position the function to management.

How are the Services Positioned Relative to the Audience or Client? Positioning development services with your clients is closely related to the service concept and norms. It is important for the CLO and the development staff to have a clear idea of the image they want to create within the organization. In describing the business of the function, the perception of its various stakeholders is essential. What image does the function portray — business partner, methodology-driven, ivory tower? What do these labels mean in the context of the company? How, when and where is the image communicated to company leaders and to other function clients? Knowing where the organization lies, in terms of its positioning, is valuable for creating action plans for future learning function management.

How Does the Function Define and Work Toward Efficiencies in its Services? Being able to articulate planned and implemented efficiency measures aids the overall analysis. In many learning organizations, it also can yield opportunities for improvements the new CLO can use to build goodwill and “capital” for the function in the company. At the surface, the CLO should look for and implement things such as clear, usable methodologies and leverage from productivity tools. On a deeper level, CLOs should be able to describe the general use of standards to guide levels of effort. CLOs also should look for management actions that support the productive questioning and improvement of processes at all levels of the function. How things get done — and how well they are done — is important. How much scrap or rework is expected? How are inputs assessed and managed? How well are staff members’ skill sets and work assignments matched? Are the subject-matter experts generally the right ones, and do they have sufficient time dedicated for the projects? When integrating the kinds of information gathered for description related to efficiencies, there will be natural links into the products and services (how standardized can/should they be?) and into the operations infrastructure (what will it support?).

At What Levels are the Different Aspects of the Function’s Operations Infrastructure Working? Operations infrastructure is a broad, complex area to describe. Its tentacles reach into everything from the processes for long-range planning to HR strategies to financial systems to publishing courses and supporting the delivery systems. As the new CLO analyzes the operations infrastructure, the form in which these various operating systems exist, how refined they are and how well they are implemented, he or she takes the description forefront. As these elements of the operating infrastructure are implemented, they affect and are affected by the other aspects of the business of the function. Thus, teasing out, looking at and describing the integrative and disruptive results that arise from the workings of the operating infrastructure are keys to the overall integrative interpretation of the business of the function.

In What Ways are the Function’s Operating Goals Balanced with the Delivery System and Client Goals? A training function will have various operating goals. For example, if the funding approach is a “fee for service” model, there might be specific goals related to cost recovery and contribution to overhead, similar to those that might exist in a nonprofit. In another organization, the goals might focus more directly on the services delivered relative to business unit size or other factors. In addition, the delivery system and clients receiving services through it will have goals and expectations that must be balanced with the operating goals. For example, clients might expect short development cycles, a high level of product quality or high levels of service responsiveness that have to be balanced. The overall goal in this part of the balancing act is to deliver value, where value is defined as the provision of a level of service (quality, volume, etc.), that is, in the mind of the client, commensurate with the costs incurred (time, effort, aggravation, money, etc.). Describing the “value equation” for the function in its unique context is a key element to understand the function’s relationship to the company and its key clients. The new CLO will want to understand if and how stakeholders perceive the value of the learning function.

What is the Delivery Strategy, and How are the Delivery Systems Structured? The CLO and the entire learning staff need to understand the most critical event in the business of the function: the moment when the service is consumed in an interaction with the function’s client. The delivery systems are the environments in which the service is provided and consumed. For learning development services, the delivery system is usually at least one employee on a team who provides the services to the sponsor/buyer at a company location. Describing the delivery system involves looking at the qualifications and skill of the team, its structure and the methods and norms used to deliver the service, as well as the facilities in which the services are offered.

The CLO should ask, among other things, “What are the capacities, the expectations and the quality controls that are in place? How is the delivery strategy consistent and inconsistent with other operating and client-oriented goals?”

These descriptions will integrate with the factors that create client satisfaction and drive an understanding of that most critical moment of consumption.

How is Client Satisfaction Understood, Monitored and Managed?
Every CLO knows client satisfaction and client loyalty are either created or destroyed in the interaction of the client with the delivery system. Every interaction either strengthens or weakens the bonds between the training function and its clients. This interaction involves all points of contact. Interacting with an executive assistant to schedule an appointment, making a call to a developer to get a clarification, meeting with a design team and other similar points of contact provide the opportunities to create satisfaction. When anything goes wrong, how the problem is handled can lead to greater satisfaction than if no problems are encountered. The new CLO needs to understand quickly the tenor of such interactions, clients’ expectations, how both the interactions and the expectations are managed and monitored and in what ways client feedback is obtained and used.

Analysis processes ebb and flow as companies and learning functions change. Many parts of the description, however, remain stable. The interpretation of the interactions, how all these elements integrate, is a challenge every new CLO faces upon taking the position. The analysis is critical for its value in learning the organization and for the advantage the information has in making the initial changes and improvements that should mark the installation of a new CLO. Following a pattern of questioning can help ensure new CLOs focus on the necessary aspects of the learning function’s business.

Oliver W. Cummings is a manager of custom development programs at Riverside Publishing. He can be reached at editor@clomedia.com.

Learning Solutions
[7] The Science of Corporate Learning
Donalee Markus, Ph.D. This article can be seen with figures through the e edition at  Chief Learning Officer   January  2007
Many corporate learning programs might sound effective in theory. But in order to positively deliver in reality, learning leaders need to understand — on a scientific level — how learners learn.

Productivity
[8] Managing Learning Function Performance
Oliver W. Cummings  This article can be seen with figures through the e edition at  Chief Learning Officer   January  2007
It's easy to say a well-designed, balanced performance measurement system that drives appropriate stakeholder behaviors greatly reduces the likelihood of function failure. However, designing and executing such a performance measurement system is easier said than done.

Tactics
[9] 12 Unavoidable Truths About E-Learning
Kenneth Carlton Cooper   This article can be seen with figures through the e edition at  Chief Learning Officer   January  2007
Many organizations ignore basic realities about technology and learning, resulting in e-learning programs that are time-consuming to create, expensive to produce and deploy and don't change behaviors in the intended way. Building effective e-learning takes an understanding of 12 unavoidable truths.

Human Capital
[13] Building Bridges: Lessons in Working with Business Units
Michelle Page-Rivera, Ph.D.This article can be seen with figures through the e edition at   Chief Learning Officer   December 2006
When working with business unites, the main function of the learning department is to best serve the needs of learners and the overall organization. To accomplish this, many learning organizations employ one of three strategies: centralized learning, decentralized learning, and hub-and-spoke learning.

Learning Solutions
[20] CLO Competencies: The Path for Future Learning Leaders
James J. L’Allier, Ph.D. This article can be seen with figures through the e edition at  Chief Learning Officer    August 2006
Many of today's CLOs are between the ages of 45 and 55. As this generation retires and the next one moves in, what work experiences and competencies will these new learning executives need? What is the definition of the future CLO?