Naturalistic Evaluation of Distance Learning Courses |
VOL. 5, No. 1, 38-52
This paper examines the characteristics of the naturalistic paradigm for enquiry. It discusses the way these characteristics can be operationalized for evaluations of academic and student support for distance education courses. Naturalistic enquiry is illustrated by reporting an evaluation of the administrative and academic support systems for a distance education course for the textile and clothing industries. The methodology is compared to that of the more commonly employed rationalistic paradigm. The advantages and disadvantages of the two paradigms for this type of evaluation are discussed.
Cet article considère les caractéristiques du paradigme naturaliste d'enquête. Les auteurs y discutent la manière dont on peut se servir de ces caractéristiques pour procéder à l'évaluation du soutien académique et administratif apporté à l'étudiant dans les cours de formation à distance. Ils illustrent l'enquête naturaliste en se basant sur une évaluation des systèmes de soutien académique and administratif inclus dans un cours de formation à distance des industries textile et du vêtement. Ils comparent la méthodologie employée à celle, plus courante, du paradigme rationaliste. Finalement, ils évaluent les avantages et les inconvénients des deux paradigmes pour ce genre d'évaluation.
The aim of this article is to outline and justify the use of a naturalistic approach to the evaluation of the academic and student support for a distance education course. This is done by comparing the rationalistic and naturalistic research paradigms, and then showing how the naturalistic approach was operationalized as an evaluation procedure. The evaluation of a certificate level distance learning course for the textile industry was conducted using a qualitative methodology which can best be described as a naturalistic approach. The method is described in detail, and direct quotation of student responses is used to illustrate the strengths of the evaluation.
When distance education courses have been evaluated, the procedures have most commonly followed a rationalistic paradigm (Dodds, Lawrence, & Guiton, 1984; Kember & Dekkers, 1987; Osman & Wagner, 1987). Owens (1982, p.4) succinctly describes the main characteristics of the rationalistic paradigm:
For distance education evaluation purposes this paradigm is normally operationalized by investigators formulating a series of hypotheses about aspects of the supporting infrastructure which might need improvement or are worth investigating. A quantitative instrument is then devised to examine the hypotheses. Likert scale items are often used to ask students to respond to statements.
Owens (1982, p.6) uses a similar six point format to characterize the alternative naturalistic paradigm of inquiry.
In this paper we will first describe our method, then we will examine each of these characteristics and explain how they were operationalized to evaluate student and academic support for a distance learning course on textiles and clothing. The approach is compared to the more commonly employed method compatible with the rationalistic paradigm. Data from the evaluation is used to illustrate the characteristics of naturalistic inquiry.
Before starting our discussion of the six characteristics of naturalistic inquiry, we feel it important to point out that the naturalistic and rationalistic paradigms are not discrete, mutually exclusive entities. They are more realistically regarded as the opposite poles of a continuum. Research projects which employ both qualitative and quantitative methodologies, for example, would fit somewhere between the two poles. We make this point because the evaluation we undertook should not be classified as a pure version of the naturalistic paradigm. The evaluation was, in fact, part of a larger research project which employed both qualitative and quantitative methods of data collection. Other parts of the overall research programme used methods which included the administration of questionnaires and the collection of students' performance data.
Before comparing the rationalistic and naturalistic paradigms we will first describe the method used for conducting the interviews for the naturalistic evaluation.
Of the 537 students enrolled in a textiles and clothing course at Hong Kong Polytechnic 12 were randomly selected to participate in this study. There were 6 males and 6 females, 5 of whom were third- year students, 4 were second-year students, and 3 were first-year students. They were contacted by telephone and invited to come for an interview in the evening. As most of the participants were employed by busy garment merchandising firms, their eagerness and willingness to attend a 30 minute interview was surprising. They would request another day if the time arranged did not fit their time-table.
The rationalistic paradigm has developed rigorous statistical procedures for estimating an appropriate sample size and quantifying the error margin in results derived from a sample of a particular size. Such precision is inappropriate for the naturalistic paradigm. Interviews and investigations might continue until a reasonably consistent pattern appears to emerge. In practice the data collection can be restricted by the resources available to conduct inter-views and the ability to analyze the resulting transcripts. This illustrates one of the fundamental differences between the rationalistic and naturalistic paradigms. In rationalistic studies, a relatively large sample is used to investigate a small number of variables, whereas the opposite is the case in naturalistic studies (Benson & Michael, 1987). Relatively smaller sample sizes are typical of qualitative research. Many such studies have been widely accepted and acclaimed in recent years, such as the research efforts of Mintzberg, whose pioneering thesis was a structured observation of just five business managers (reported in Mintzberg, 1983).
The interviews were conducted individually, usually after regular office hours, in one of the seminar rooms at the Polytechnic. The interviewer, the project's research assistant, used a schedule that had been prepared by the research team. All interviews were conducted in Cantonese, except one which was conducted in Mandarin, and all were tape-recorded. The interviews ran from 30 to 45 minutes.
Before starting, the purpose of the interview and its usefulness were explained. Once the interviewee was happy with the arrangement, the interview would proceed. The interview schedule consisted mainly of semi–structured questions. If any item was unclear, the interviewer would give examples, being careful in doing so to avoid biasing the participant's response.
The third- and second-year students participated more actively than first– year students who had less experience with distance education. The participants pointed out the strengths and weaknesses of the course, and what should be improved - their responses seemed open and uninhibited. This might be due to the maturity of the participants. Unexpected and helpful comments were also made about aspects of the course which were not covered in the interview schedule.
After the interviews, the audio tapes were transcribed and written English protocols were prepared. The participants' responses were then sorted under several broad headings, such as education, tutorial support, and administration. For the final analysis, two members of the research team made separate, independent analyses of the transcripts. Similar comments from students were grouped together to determine if any consensus existed. The outcome of the two analyses were then studied by the research team. Considerable overlap was found, giving strength and confirmation to the conclusions reached. For discussion of the principles of coding, see Grove (1988).
A number of techniques were used to assess the validity of the findings. The principal techniques, which have been derived from naturalistic research methodology, were triangulation and peer consultation.
Triangulation (Guba, 1981) concerns the cross-checking and verification of data through the use of different information sources. This can include a variety of data sources, investigators, theoretical models and research methods. Obviously, stronger conclusions can be drawn from comments made by more than one interviewee. Also, conclusions drawn from the interview data could be compared to the results obtained from a separate quantitative questionnaire which was administered to over 400 students. For example, on the basis of comments made during the interviews, the conclusion was drawn that feedback on assignments was often deficient. The validity of this conclusion could then be assessed by examining responses made to questionnaire items that addressed this concern.
Peer consultation (Owens, 1982) aims to give researchers opportunities to check and test their findings as they unfold. The consultation should be amongst qualified, interested peers who are able to adequately assess the conclusions that are being reached. As already mentioned, a safeguard built into this study was the decision to have two members of the research team analyze the student interviews separately in order to group opinions, conclusions and queries. These analyses were then distributed to other members of the team for comment and feedback. The consultation continued until the team was satisfied that the analysis accurately reflected the views expressed in the interview transcripts.
This section examines each of Owen's (1982) six characteristics of naturalistic enquiry and shows how the choice of paradigm influences the nature of the evaluation.
In a rationalistic inquiry, each variable or facet of a problem is isolated and then used to explain a phenomenon or draw a conclusion. The context (e.g. the student's family or work background, the personal reasons why a particular student seldom attended the tutorials) in which the variable is embedded is isolated or ignored.
In a naturalistic enquiry, on the other hand, the phenomena under investigation must be studied as a whole and cannot be teased out of context. The aim of the interview, then, is to provide a more comprehensive picture of each aspect of the study.
In this study, partially structured questions were used in the interviews. An example of the type of question asked was: Do you attend the tutorials? The question was structured to encourage the interviewee to focus on a particular topic, but it was open-ended so that the respondent could elaborate on points of interest. Interviewer responses were limited to such comments as, "Um, um, ... can you tell me more about it? ... OK. ... Yes, ...."
The following extract from one interview session illustrates the richness of the answers to the interview questions.
Yes, I attend most of the tutorials. It certainly helps when I meet good tutors who can foresee the possible problems met by the students, who can explain the subject matter clearly, solve the kinds of problems that the students come across, and point out the important ideas in the course. Unfortunately, there are a few irresponsible tutors too. For instance, one tutor asked the class whether anyone had any problems, if not, he said the class was dismissed. It really was a waste not only of our time but also our money. I am not sure whether the authority concerned has indicated to the tutors what their roles are. Again it is a waste of time and money for the tutors to distribute the course materials during the first tutorial in each semester. In fact, the distribution of course materials could be done by the clerical staff in the administration office. We could be informed to collect them any time during office hours. We are not little children any more. There is no need for the tutors to explain matters such as assignment due dates, exam schedules, etc. as all the information has been given to us on the information sheets. I hope the tutors can make good use of the tutorial periods, otherwise they are useless.
In a naturalistic study it is acknowledged that it is impossible to eliminate the interaction between the inquirer and the subject. Very often the inquirer in a naturalistic inquiry is an integral part of the phenomenon under investigation. For example, in an evaluation of a distance learning course, the tutors, who are part of the course, may take the role of inquirer. This point was well taken in the interviews. In this study, the interviewer was a research assistant who was not a tutor or administrative staff member of the course. This reduced possible interaction between the interviewer and the subjects. Nevertheless, some of the subjects under investigation treated the interviewer as part of the administration of the course, as was evidenced by the fact that a few subjects submitted assignments to the interviewer!
Since an interviewer was used as an instrument for gathering data, the research study is "willingly trading off some objectivity and reliability (in the rationalistic sense) in order to gain greater flexibility and the opportunity to build upon tacit knowledge (a feature that pen-and-pencil or physical instruments can never have)" (Guba 1981). Flexibility in the collection of data was reflected in the fact that subjects could choose a time and place for the interviews which were convenient to them. The use of a pleasant and friendly interviewer enabled the interviews to be conducted in a free and comfortable atmosphere. This method has the advantage mentioned earlier. During the interviews, possible reading or language problems, which are common in the administration of questionnaires (Harris, Bell, & Carter, 1981), were avoided because the interviewer could always explain or rephrase a difficult question. As mentioned, the interviewer tried not to influence the interviewees while they were responding.
The interview data cannot be used to make generalizations beyond that bounded by the study. When the findings from the interviews are used, they always relate to the particular context. For example, the following comments about the timing of examinations in January apply only to the Fashion and Clothing Manufacture distance learning course.
The exam schedule in January is another headache to us as well. From August to January is the critical period for garment merchandisers. I can hardly find enough time for my revision of course work. We are usually free in the middle of the year since the European market is quiet during that time.
Again, every December is the busiest month for the garment business and unfortunately the exam usually falls in the following month, i.e., January. Because of the big volume of business, we find that it is very hard to find adequate time for revision before the exam. I have written to the Polytechnic and suggested that exams not be scheduled in January but it seems to me that no action has been taken so far.
The findings and recommendations have limited generalizability to other distance learning courses in Hong Kong, but were not meant to generate axioms applicable internationally.
Enquiries based on the rationalistic paradigm have normally used quantitative methods, whereas proponents of the naturalistic paradigm have tended to use qualitative techniques. Indeed the tendency has become so strong that the paradigms have been equated to the methodologies. However, as Guba (1981) points out, both paradigms can accommodate either methodology.
Quantitative evaluation procedures in a rationalistic paradigm commonly use items with Likert scale responses, to examine students assessments of services provided. For example, a quantitative evaluation of assignment marking might look something like . As such questionnaires can be administered and processed with economy of effort, reasonably large sample sizes can be employed.
If a majority of students agree or strongly agree with a sample statement, the evaluators normally would decide that written assignments were not a problem, though problems with the marking of individual tutors could be masked by a majority response. If the majority of students disagree with a statement it clearly indicates that a problem exists but it may not indicate the nature of the problem. The scale can be more difficult to interpret, however, if responses are distributed across the scale or if they gravitate toward the centre.
The qualitative evaluation report described in this paper contains quotations which may have coincided with a distributed or neutral Likert scale response.
The comments on returned assignments are helpful but it still depends on the tutor. Some tutors are terrible, the comments made are just `Try your best,' - that's all. Some of the bad comments are as follows: `You have answered 4 options but there are still 2 more.' The tutor didn't give the answers. I wonder that I will know them when I see them in the book. I wouldn't have missed those options if I had known what they were beforehand. Can I suggest that model answers be provided?
Unlike before, now more comments are made on the returned assignments which are quite helpful for our examinations later.
A marked difference in the performance of the tutors, or an improvement over time, could have led to a neutral response, yet clearly the work of some tutors is quite unsatisfactory. The first comment above illustrates the rich and specific data which can be obtained from interviews of this type.
Evaluations based on the rationalistic paradigm usually test hypotheses. Researchers evaluating distance learning courses by quantitative methods typically frame hypotheses on issues which they think might concern students. Typically these hypotheses are then translated into Likert scale items similar to those in Figure 1. A potential problem with this procedure is that the hypotheses and the generated items may completely miss a strong concern of students. Alternatively, student responses to an item may indicate concern in a particular area without pinpointing the precise nature of the problem.
Interviews which are semi-structured, on the other hand, allow students to raise issues which concern them. The interviewer may direct a student toward the topic of tutors or administration, for example, but the student is then free to raise any issue or concern related to those topics. The students, therefore, can raise issues which might not have been predicted by the experimenters. The two quotations below illustrate this point.
The marked assignments are returned during the tutorial classes. Those who didn't attend the tutorials need to come to the administration office to collect their assignments during office hours, which I think is quite troublesome. We can only come to collect the marked assignments during our lunch hour which is usually the same as the Poly's lunch hour. The administrative staff are impatient, not helpful to me when I come to collect my assignment during lunch time. The administrative staff should understand our position as distance learning students. Sometimes we really need to take a half-day casual leave just for picking up a marked assignment. One thing I hate is the half-year renewal of the student card. Renewal is only possible during office hours, which is quite troublesome to us. We need to take a half-day casual leave when the time of renewal comes. Should the system of renewal be modified?The issue of renewal of the student card would certainly not have been foreseen as a questionnaire item. The problem of office hours is also unlikely to have been envisaged.
As naturalistic designs start without hypotheses, theories (or in the case of this evaluation - recommendations for change) emerge from the data. Glaser and Strauss (1967) describe the theory as being "grounded" in the collected data. The way the analysis procedures were operationalized for this evaluation project were described in the section on analysis.
There is a conventional format for reporting rationalistic studies: hypothesis tested, method, experimental results, discussion of results. Naturalistic enquiries do not commence with the formulation of hypotheses and usually have quite different types of data so require different formats for the report. Descriptions of the context tend to be longer and the reports are usually replete with quotations. Theories or recommendations are usually portrayed as emerging from the quotations, which are typical of observed classifications. The reports are usually longer than typical reports of rationalistic designs. The report given to the department running the evaluated course contained a description of the interview method and the analysis procedures. The remainder of the report consisted of observations derived from the analysis illustrated by selected quotations. Recommendations for modifying the support systems were derived from the observations.
For a pure rationalistic enquiry, it should be possible to specify the research design before any data is collected. The naturalistic paradigm anticipates developments to both the research design and the topic of investigation as the study progresses.
In this particular evaluation the method for arranging interviews and the questions asked were monitored and modified after the initial interviews. The schedule was subsequently changed several times as issues and concerns became apparent. The course also underwent changes. One recommendation related to the quality of assignment marking by tutors. Before the report had been completed a large influx of part- time tutors were hired. These tutors all participated in a workshop, based on a distance learning package for new tutors (Kember & Murphy, 1988). The performance of tutors now needs to be re-examined in light of the new tutors who have passed through the new induction process.
When the naturalistic paradigm is followed, enquiries do not have clearly defined start and finish points. Examination of the course can continue, to a greater or lesser extent, as long as it operates. In this investigation the number of interviews was not pre-determined. Data was monitored as it was collected and interviews continued until reasonable levels of consistency became apparent on major issues - and the pile of paper became too high.
A clear advantage of the naturalistic paradigm is that it enables the respondents to focus on issues that are of real concern to them. Answers to the questions can have great depth and can be illustrated with specific examples. The researcher can, therefore, gain great insight into the problem being evaluated. Furthermore, to the extent that it can avoid possible reading or language problems of the interviewees - any unclear questions are clarified before they are answered - this method is considered more reliable than the questionnaire method.
Using the interview method, one can gather a large amount of information about an aspect of a problem in a short period of time, although only from a small sample. Preparation of transcripts and systematic analysis, however, is time-consuming. Other disadvantages are that planning and administration are also time-consuming, and that the interviewees' attitude may be influenced by the interviewer (Harris, Bell, & Carter, 1981).
A further disadvantage of qualitative data is that the researchers can be less sure of the generalizability of the findings, even within the programme. In the previous section it was concluded that the results of this evaluation study were not necessarily generalizable to other courses in other institutions. The quotations above raise the issue of the extent to which inferences from the quotations about tutors could be generalized to all tutors. In this particular instance it is clear that there is variability among the tutors.
Given the small sample used in this study, the question of internal general-izability must also be considered. If, in large quantitative studies, the majority of subjects respond negatively to a survey item, the message is difficult to ignore. The same degree of certainty cannot be assigned, however, to inter-view data obtained from a small sample. Adherence to a strict analysis protocol, as discussed in the final section, can help counter this problem, as can the nature of the quotations included in the report. Gripping comments which are rich in detail are difficult to ignore.
For some years evaluators have been urged to broaden their approach to research. The rationalistic paradigm had become so entrenched that proponents of the alternative naturalistic paradigm took a position diametrically opposed to the prevailing values. Parlett and Hamilton (1977), for example, equated the rationalistic paradigm to an "agricultural-botany" paradigm in an article which urged evaluators to adopt an alternative illuminative approach. More recently, Guba and Lincoln (1982) have been keenly advocating and conceptually strengthening naturalistic inquiry. The paradigm has now become more widely accepted and articles regularly report educational research following the naturalistic paradigm.
Distance education researchers have also been urged to widen or change their perspectives. Morgan (1984) noted the under-representation of qualitative methodologies, and described how such methodologies can be used to advance knowledge on how students learn at a distance. Further, Minnis (1985) espoused the application of the qualitative methodologies used by anthropology and sociology (ethnography, case study and grounded theory) to distance education, while Murphy (1986) briefly outlined the ways that single-case experimental design might be used by distance education researchers.
This article is part of the effort to redress the imbalance in the use of methodologies. It also broaches the paradigm problem, justifying and strengthening the use of qualitative methodologies by explaining the links between alternative methodologies (qualitative/quantitative) and paradigms (naturalistic/rationalistic). We believe that the results reported in this article aptly illustrate the power of the naturalistic paradigm, especially when teamed with an appropriate methodology. Such an approach not only enhances the credibility of the paradigm, but also builds in methodological rigour, particularly with respect to the reliability of procedures.
We would not wish to enter into the "paradigm war" (Gage, 1989) forced upon the early proponents of the naturalistic paradigm. To advance the acceptance of the naturalistic paradigm they vigorously attacked shortcomings of the rationalistic paradigm, so two mutually exclusive schools formed. We do not believe that one paradigm is inherently superior to the other but that both have an appropriate context. We believe that a judicious combination of quantitative and qualitative methodologies will provide useful results. It has been argued that such a combination, from the same setting, can generate more powerful analyses than either approach could have generated alone (Miles, 1983).
Smith and Manning (1982) suggest that qualitative research is better for initial exploratory research into unknown or unfamiliar phenomena. It enables the researcher to develop concepts, conjectures, interpretations and theories empirically grounded in the investigated system. Quantitative research can be more focussed. It can be used for rigorous analysis and corroboration of the insights and theories of the exploratory study. Smith and Manning (1982, p.
xiv) show the process of cumulative social science as a progression from paradigms, through qualitative then quantitative research to formal theorizing. The presence of feedback loops allows for iterations in the progression.
The evaluation described in this paper is a component of a larger research project examining student progress in distance education courses. To identify the important constructs and to derive theories relevant to this complex issue we are utilizing both qualitative and quantitative methods.
Benson, J., & Michael, W. B. (1987). Designing evaluation studies: A twenty-year perspective. International Journal of Educational Research, 11(1), 43–55.
Dodds, A. E., Lawrence, J. A., & Guiton, P. (1984). University students' perceptions of influences on external study. Distance Education, 5(2), 174–185.
Filstead, W. J. (1970). Qualitative methodology. Chicago: Markham. Gage, N. L. (1989). The paradigm wars and their aftermath. Educational Researcher, 18(7), 4–10.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago: Aldine. Grove, R.W. (1988). An analysis of the constant comparative method. Qualitative Studies in Education, 1(3), 273–279.
Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic enquiries. Educational Communication and Technology Journal, 29(2), 75–91.
Guba, E. G., & Lincoln, Y. S. (1982). Epistemological and methodological bases of naturalistic inquiry. Educational Communication and Technology Journal, 30(4), 233–252.
Harris, N. D. C., Bell, C. D., & Carter, J. E. H. (1981). Signpost for evaluating: A resource pack - Part 2. Collecting and analyzing information. London: Council for Educational Technology.
Kember, D., & Dekkers J. (1987). The role of study centres for academic support in distance education. Distance Education, 8(1), 4–17.
Kember, D., & Murphy, D. (1988). Tutoring distance education courses. Hong Kong: Hong Kong Polytechnic.
Miles, M. B. (1983). Qualitative data as an attractive nuisance: The problem of analysis. In J. Van Maanen (Ed.), Qualitative methodology. Beverly Hills, CA: Sage.
Minnis, J. R. (1985). Ethnography, case study, grounded theory and distance education research. Distance Education, 6(2), 189–198.
Mintzberg, H. (1983). An emerging strategy of `direct' research. In J. Van Maanen (Ed.), Qualitative methodology. Beverly Hills, CA: Sage.
Morgan, A. (1984). A report on qualitative methodologies in research in distance education. Distance Education, 5(2), 252–267.
Murphy, D. (1986). Experimental design: The case for single case. ICDE Bulletin, 10, 20–23.
Osman, R., & Wagner G. A. (1987). New Zealand management students' perceptions of communication technologies in correspondence education. Distance Education, 8(1), 47–63.
Owens, R. (1982). Methodological rigor in naturalistic inquiry: Some issues and answers. Educational Administration Quarterly, 18(2), 1–21.
Parlett, M., & Hamilton, D. (1977). Evaluation as illumination : A new approach to the study of innovatory programmes. In D. Hamilton, et al (Eds), Beyond the numbers game. London: Macmillan.
Smith, R. B., & Manning, P. K. (1982). A handbook of social science methods, volume 2: Qualitative methods. Cambridge, MA : Ballinger.
This project was supported by a research grant from the Hong Kong Polytechnic.
David Kember, Tammy Lai, David Murphy, Irene Siaw, Julian Wong, and K.S.Yuen.
ETU
Hong Kong Polytechnic
Hung Hom
Kowloon, Hong Kong.