online learning

Testing Online Learning Materials

Testing online learning materials involves multiple levels of quality assurance. In general, these include testing done to ensure the fidelity of the media used to present the material and testing done to ensure that the material presented leads to the desired student outcomes. Ensuring the fidelity of the media is the easiest of these to do. There are two basic levels of media fidelity testing. First is at the point of media creation. This level of testing is done to ensure that the media that has been produced is complete and of an acceptable quality. This testing is most often accomplished by the creator of the media. In group settings, this would be the person who owns the gold version of the media prior to release of that media to the actual use environment. The second level of media fidelity testing is more complex and involves testing the media in as close to its release form as possible. This usually consists of having someone review the materials as they will be situated when the consumer of those materials uses them. This allows the tester to ensure that links work as desired and that the media transferred properly to the location from which it will be used. A critical test of media fidelity at this level is basic validation of any coding used to create or present the materials. This can be done through the use of any one of a number of web-based validation checkers. There are validation tools that will check HTML code, CSS coding and ADA compliance. The degree of validation that you pursue generally increases as funding and scale of the release. Well- funded projects that expect to release the media created to a very wide audience will typically do the most validation of their product. This is especially critical when the product will be released at a level where it can no longer be adjusted by the creator. By this I mean that in some releases, the materials that have been developed don’t reside on a server that is controlled by the developer of the original media. In such cases, it is very difficult to change the materials once they are released. Small-scale developments, for example, materials developed for this class typically stay under the control of the developer so a lower level of validation is usually done because it is easy to correct any errors that may be uncovered in beta testing by actual users. This is another way of saying that it is often easier to correct errors in the released version of the media than it is to do the level of testing required to locate all possible errors in the materials. A sophisticated level of media fidelity testing examines the media under a reasonable set of circumstances that end users are expected to encounter. For

example, it is a very good idea to test media on multiple operating systems and if appropriate, in different web browsers. Again, there are a variety of web tools available that will provide a way for developers to understand how the materials they create work on different operating systems and different software installations. Nothing, however, beats actual testing of materials on real machines running different browsers and with different software installations. It is amazing how differently something can work on two different machines based on simple factors such as software and even hardware differences between the two machines. On one of the larger development projects I have worked on, a project funded at over a million dollars per year, I made sure to keep a variety of older machines around, both Macintosh and Microsoft operating systems, with different versions of operating systems, different browsers and even different versions of browsers. What I was trying to do was replicate usual and anticipated hardware and software combinations that our ultimate end users might be using. One thing that many test procedures forget about is differences in bandwidth and network performance issues that end users may experience. For example, the media for this class is largely created at a large university that has excellent networking and bandwidth. In addition, in some cases the materials that are being tested are located on servers that are inside the university network. This all combines to ensure that users who access the materials at the university experience very fast network performance. But what happens when a user who lives hundreds or even thousands of miles away from the network that hosts the materials? A lot can happen! It is important to understand how these sorts of factors can degrade your end user’s perception of media and material quality. If possible, you should test your materials in locations as similar to your anticipated end users as possible. Another type of testing that needs to be addressed is testing to ensure that the materials adhere to guidelines established to ensure that individuals with disabilities can access and use the materials that have been created. There are both laws and institutional policy to consider here. Laws include Section 504 of the Rehabilitation Act of 1973 that requires employers and institutions that receive Federal money provide equal access to information, activities, program and services to all individuals with disabilities and the 1990 Americans with Disabilities ACT extends the equal opportunity provision to private organizations. Typically, larger institutions will write policy to comply with the legal requirements. For example: The University of XYZ is committed to providing equal opportunity for persons with disabilities. This commitment includes complying with the Americans with

Disabilities Act of 1990 (ADA) and Sections 504 and 508 of the Rehabilitation Act of 1973. In addition, all of the University's websites and online courses will comply with the web page design standards established by the World Wide Web Consortium (W3C). The university should adopt a plan, whose adoption should not extend past three years, for bringing all Internet sites and on-line courses into Priority One compliance with W3C guidelines (http://www.w3.org/WAI) or to exceed them. Faculty members posting instructional materials online or utilizing course management software should plan to assure their accessibility. In general, these laws and policies require that all instructional materials be accessible to individuals with disabilities. Typical disabilities and their accommodations include:

Color Blindness High Contrast Avoid Red-Green-Blue Combinations Don't Use Color to Convey Meaning Make Use of a User Changeable Font Text Version

Low Vision – No Vision High Contrast

Make Use of a User Changeable Font Navigation Requirements Configurability – Screen Reader Compatibility Text Version

Seizure Avoid Strobes Flickering Images Fast Animations High Contrast Close Lines (optical illusions)

Auditory

Alternatives to Audio Text Version Captioning Videos and Slide Shows with Audio (Multimedia)

Motor

Navigability Issues Keyboard Control Instead of Mouse Internal Navigation to Skip Through Long Sections User Configurability

Typically, accommodations are provided first by following W3C guidelines for the use of CSS and other issues. The next level of accommodations involves creating alternative representations of materials, for example always providing a text version of any audio resource or close captioning any slide show or video. All of these types of fidelity testing tend to be technical in nature. That is to say, this type of testing, and if needed, changes to materials require a certain level of technical understanding and technical development skill. The next type of testing deals with the content. This second type of materials testing is related to the instructional quality of the media. This can be a very complicated matter. In general, this type of testing occurs in many stages along the development cycle for the materials being created. The key to this type of testing is to ensure that the materials that are created result in the desired student outcomes. The first thing that has to be in place for this type of testing to work is a clear understanding of the desired student outcomes. Most often, learning outcomes are expressed as objectives that describe an observable trait or behavior in the learner. Typically, these objectives are established in the early stages of the instructional design process and are carried forward to the materials development cycle. It is important to assign specific objectives to specific materials, and often to specific sections or elements of those materials. This assignment of objectives is what lets you test, and modify materials effectively. The first level of instructional quality testing is typically done long before the media is created. The first level of instructional quality testing is often a review of the media design plan by subject matter experts. A with most things, the larger and better funded the project the more extensive the scrutiny and/or testing will be. In one project I worked on, we employed two levels of subject matter experts to review materials before they were ever created. The first level was a high level content review. This review established quality at the programmatic level by examining proposed topics in a domain of study to ensure that the topics were consistent with an expert’s opinion of what should be taught within that domain of study. The second level of review was done by another group of subject matter experts who reviewed outlines of each presentation that was to be written to ensure that the outlines addressed the essential content issues within the identified topic. After this review, a writer would be contracted to produce a transcript using the approved outline. In smaller developments, the creator of the materials will often serve as their own reviewer or the developer may work with another developer to cross check each other’s work.

It is important to do as much quality checking as possible before media development occurs because typically, once the media is in development, and especially after the media is created, changing that media is more expensive and time consuming. That said, it is important to test the materials you create in as close to the actual use setting as you can manage. In most cases, this is done in a beta test with actual students. Careful comparison of learning outcomes resulting from the use of the materials gives you the information needed to assess whether or not the materials are working effectively. Knowing what objectives are addressed by which components in your media or materials gives you the information you need to make changes which may be needed to achieve the learning results you intend. This presentation has hopefully given you some insight into two different types of instructional materials testing. First, the more technical media fidelity type of testing and second the more content focused instructional quality testing. Both are essential to a good product. In general, the larger the development, the greater the funding level or the wider the materials are expected to be distributed the greater the care taken in each of these kinds of testing. It is safe to say that almost any level of development can benefit from even more testing than has been done!