9.2: Quality in E-learning

The Problem of Quality

Certain e-learning and pedagogical innovations have not succeeded in meeting a number of promises (Salmon, 2005), and have created confusion between the mere supply of information and actual knowledge-building and training (Barbera, 2004). Projects aiming at supporting e-learning environments such as UK eUniversity, NYU Online, Scottish Knowledge, Universitas 21, Global University Alliance (Garret, 2004), as well as a number of European corporate learning projects (e.g., StarScience, Dunes, Adapt-IT, Teachers-in-Europe, POLE STAR) have failed to realize many of their goals. However, the collapse of such initiatives does not indicate the failure of the e-learning concept per se, but rather the lack of quality. For example, lack of planning and marketing were the major reasons for the UKeU failure (Garrett, 2004.) The questions that arise include: What constitutes quality in e-learning? Why is it important? Are there ways we can ensure e-learning quality?

In general, quality refers to fitness of purpose. In e-learning, quality refers to learning (Stephenson, 2005), something excellent in performance (EFQUEL, 2005). In particular, quality in e-learning means providing the right content at the right time, enabling learners to acquire knowledge and skills and apply their learning to improve their performance, whether as an individual or within an organizational framework (ASTD & NGA, 2001). Stephenson (2005) proposed that quality depends on its interdisciplinary nature, and the identification of quality factors for a given environment depends on the chosen perspective. As there are two essential levels—the pedagogical and the operational—the target for return of investment must therefore be viewed as long term.

National bodies and international organizations have now developed principles, guidelines, and benchmarks to describe quality based on the international developments in the field (QAA, 1998; CHEA, 2001; USNEI, 2001; ISO-9000, the Benjamin Franklin Institute, 2001; EFQUEL, 2006). Furthermore, importance is also attached to national standards resulting from the globalization and universal access of learners as customers and taxpayers. For example, in Europe, there were efforts for regionally harmonized systems (see Bologna Declaration, European Ministers of Education, 1999) and Quality Assurance (QA) and accreditation systems developments. Brajnik (2001) proposed that a quality model seeks ‘quality’ by:

• understanding, controlling, and improving a product or a process;
• identifying problems or performance bottlenecks, base-lines, and timescales, and,
• comparing these for progress assessment, as well as for distinguishing certain attributes from others.

This method for developing and documenting a quality model suggests the production of a complete and consistent set of quality requirements (Firesmith, 2003). Attempts to provide such quality frameworks were conducted by European organizations but they have yet to be fully tested.

The European Foundation for Quality in E-learning (EFQUEL) was established in June 2005 in order to provide a coherent framework of quality factors for all European organizations. Its mission is “to enhance the Quality of eLearning in Europe by providing services and support for all stakeholders” (Nascimbeni, 2005, EFQUEL). This means that the quality factors are explicitly connected to the provision of services and support for all stakeholders from different fields. EFQUEL has attempted to map a quality model by incorporating stakeholders’ perspective for policy makers, researchers, e-learning quality related organizations, decision-makers, e-learning users, and learners. EFQUEL conducted a European survey between 15 August 2004 and 15 November 2004 (Panorama Report, Ehlers, Hildebrandt, Görtz & Pawlowski, 2005). Of the 5,023 responses, 28 percent completed it, and only seven percent finished the two basic sections on quality of e-learning. (The low response rate may have been due to inherent difficulties of understanding and defining what e-learning quality is. It is perhaps easier to described what quality is than to define it (Stephenson, 2005).) According to the results, quality relates to obtaining the best learning achievements (50 percent) and “something that is excellent in performance” (19 percent). In detail, the Panorama Report revealed the following:

1. the importance of e-learning quality: Quality is, and will be of great importance for e-learning.
2. the need for specific frameworks: Although numerous quality strategies and concepts were used, the understanding of quality is lacking—this being conceived as an abstract rather than a concrete form. The respondents believed that they knew about quality but they showed a general lack of information on quality measures, and that deceived them.
3. quality requirements in design: Learners are both users and customers, and are seldom involved in design in public and business sectors, but design for quality needs to consider the following issues:
   • their recommendations for successful quality development. This will prevent the low level of acceptance of designs that lack user quality.
   • the inclusion of organizations’ own checklists for quality found in web resources, discussion forums and fairs;

     The above shows that designers of quality must have experience of quality and ability to meet challenges; to change and adapt, to incorporate quality strategies, and being open to creativity for entirely new forms of quality development.

4. critical awareness: Analysis and criticism of quality demands:
   • a high degree of critical awareness;
   • quality systems that reconcile the objectives of all the individuals involved;
   • quality must be seen as a dynamic process of adaptation to users’ needs, primarily those of learners.

The researchers produced a framework of processes for describing quality approaches. This framework refers to general conditions of e-learning that comprise analysis of the external context; design and production involving testing, adaptation, and release of learning resources; implementation, evaluation, and optimization; and lastly, establishment of requirements such as initiation, identification of stakeholders, definition of objectives, and analysis of needs. They stressed that “learners must play a key part in determining the quality of e-learning services”, and insisted on the integration of all stakeholders in the process. The outcome of the European efforts was the white paper ISO/IEC 19796-1 entitled How to Use the New Quality Framework for Learning, Education, and Training (Pawlowski, 2006)— yet to be fully tested.

In the UK the Government’s University for Industry has embraced a learner-centred approach, learning to be determined by the learner, for its ICT programmes. This is to be done by transforming traditional methods of learning (University for Industry, 2003). The Quality Assurance Agency for Higher Education (QAA, 2004) described quality assurance as a code of practice with conditions in place for students to achieve, as set by the institution (QAA, 1998). QAA evaluation is based on teams of academics conducting audits and processing learner reviews. (For detailed QA efforts and comparison see Parker, 2003.) The quality assurance seems to be a description of quality factors for a planned and systematic review of an institution or a program. This description determines the acceptable standards of learner-centred education, scholarship, pedagogic culture and expertise, infrastructure, and organizational strategy and vision, and ensures that these are being maintained and enhanced (Pond, 2002). In the business sector, quality of e-learning in organizations is associated with guidelines for finding and choosing quality in e-learning courses, services, and providers in the e-learning marketplace (WR Hambrecht + Co, 2000).

Because of its intangible dimensions, e-learning quality control creates challenges to contemporary research. Overall, e-learning quality appears to derive from interdisciplinary approaches on learner-centred frameworks and depends on the organization’s’ infrastructure, organizational strategy, and vision. However, working on a meta-study on e-learning, Pinelle and Cutwin (2000) reported that in real world settings only one-quarter of the articles included evaluations. Thus researchers missed the current transition from the Industrial to the Information and Collaboration Age as the Tavistock Institute had predicted in 1949 (Mumford, 1983; Dolence & Norris, 1995, cited in Parker, 2003). In fact, these changes are apparent in the ways people work, learn, and entertain themselves, which shows the need of multiple skills within an organization. Therefore, although QA processes are necessary, it is difficult to set specific QA standards in a transitional period. In this connection, a European survey on e-learning quality revealed the problem of reflecting reality, and directly associated it with e-learning instructional design (Massy, 2002).

In order to ensure quality education without empirical and systematic assessment, Pond (2002) provided a set of universal attributes (criteria). He referred to the most widely used definitions of quality, quality assurance, and accreditation, with the learner at the centre of the evaluation process. According to Pond, accreditation is the process used in education to ensure that schools, post-secondary institutions, and other education providers meet and maintain the minimum standards of quality and integrity. This would include academics, administration, and related services (USNEI, 2001). He called on the Council for Higher Education Accreditation to define quality. In its glossary for International Quality Review, quality is defined as “fitness of purpose—meeting or conforming to generally accepted standards … [Quality assurance is] … planned and systematic review … of an institution or program to determine that acceptable standards of education, scholarship, and infrastructure are being maintained and enhanced” (CHEA, 2001). That is to say, learners’ expectations have to be met or exceeded. In other words, they must acquire knowledge and skills that they did not possess before the learning experience took place. Wallace (1999) and Smulders (2002) saw the learner in e-learning as both a learner and a user, and then quality standards need to be defined in practical terms on both pedagogical and operational levels.

Quality Assurance (QA)

Quality assurance (QA) is a planned and systematic review process of an institution or program to determine that acceptable standards for learner-centred education, scholarship, pedagogic culture and expertise, infrastructure, and organizational strategy and vision, are being maintained and enhanced. This would include expectations that mechanisms of quality control for benchmarking are in place and effective. QA provides the means through which an institution ensures that conditions are such that students can achieve the standards set by that institution or other awarding body. Benchmarking provides signposts against which outcomes can be measured. Subject benchmark statements allow the academic community to describe the nature and characteristics of programs in a specific subject. They also represent general expectations about the standards for qualifications at a given level; they articulate the attributes and capabilities that those possessing such qualifications should be able to demonstrate. Benchmarking is therefore a prerequisite for quality assessment.

Quality assessment is a diagnostic review and evaluation of teaching, learning, and outcomes based on detailed examination of curricula, structure, and effectiveness. It is designed to determine whether or not the institution or program meets generally accepted standards of excellence, and to suggest further quality improvements.

Quality improvement refers to expectations that an institution will have to plan, monitor and improve the quality of its programs. In most cases, the quality assurance of an accrediting agency requires established procedures to ensure an ongoing process (CHEA, 2001). According to Pond (2002), the new educational online paradigms are learner-centred, tailored, open, collaborative, qualitative, and flexible. They may also be locally differentiated. These criteria meet a universal set of quality e-learning criteria. Online education should therefore provide:

• continuity between advertising and reality
• continuity between purpose and practice
• preparation for external credentialing/further study
• personal/professional/academic growth for the learner
• relevant
• rich, multidirectional interaction
• functional, user-friendly interface
• adequate resources for: instructors, learners, curriculum
• appropriate assessment methods/opportunities

Pond’s criteria seem to be eminently constructive for a learner’s development.

In conclusion, quality assurance, assessment, and improvement require sets of performance, benchmarks, and indicators based on evaluation tools and techniques. The latter need specific criteria anchored in quality factors. E-learning quality factors describe these systematic reviews and evaluation of principles, guidelines, and benchmarks. However, there is a problem related to labour-management issues during collective bargaining vis-à-vis quality education. It is important that to be in alignment with the international, national and organizational targets need to be in alignment. This is the major challenge.

Quality Factors

It is evident that ‘quality is easier to describe and illustrate than to define’ (Stephenson, 2005:1). Ensuring e-learning design for cognitive engagement in practice associated with outcomes is exactly what constitutes e-learning (Kalantzis & Cope, 2004; Oliver, 2005). Systems design has to ensure factors for quality at different levels and fields, micro or macro (Hedberg et al., 2002). Recent studies aim to identify quality factors. These studies are guides to good practice (Grahan et al, 2002); indicators for online teaching (Corich et al., 2004); pedagogical dimensions for computer-based education evaluation (Reeves, 1997); quality management (López et al., 2003); learners’ perspective (Ehlers, 2004); pillars for quality assurance and accreditation (Pond, 2002); and evaluation frameworks and tools (Muir et al., 2003). These studies referred to specific institutions’ QA standards, defining all stakeholders’ goals based on international, national, and organizational frameworks.

According to the International Standard Organization ISO/IEC 19796-1, QA can be ensured by:

• identifying the main quality objective for a process;
• identifying the responsible actors;
• identifying methods or instruments that can be used to assure quality; and
• designing to measure the success of the quality objective.

For example, if an organization provides short-term programming courses for groups of 20 students learning C++ in two weeks, the online teaching and learning style is quite different than it would be if the objective were to learn Greek. The system needs to meet the learner’s objectives. Another example is proposed by Parker, as four QA principles (2003):

• guaranteeing consistency in the product’s results based on long-term values;
• guaranteeing consistency in governmental and corporate education;
• guaranteeing learner-centred education;
• guaranteeing collaboration between internal and external stakeholders).

Parker believes that in order to maintain continuity and consistency it is important to define values. As mentioned earlier, collaboration between the stakeholders for a learner-centred education is the key to success. Institutions need to have a proper understanding of their monitoring operations if they are to improve decision-making and performance. This being done, they will satisfy both themselves and external agencies that they are effective in achieving aims and objectives, as well as being cost-effective and cost-efficient (Rumble, 1986).

To sum up, specific frameworks are necessary to specify quality factors and requirements fit for purpose. Collaboration between all stakeholders is critical: involvement of all stakeholders in the process of design is important: good evaluation tools and techniques ensure quality. E-learning is valuable as an added learning environment to enhance human capabilities further.