November 16, 1977

Rosemary Wilson
Office of Education
Department of Health, Education
     and Welfare
7th and D St. SW
Washington, D.C. 20202

Dear Ms. Wilson:

The critical Follow Through issue is a moral one. We have demonstrated the capacity to teach you and other educators about teaching "poor kids", turning them on, and assuring that they catch up to their middle-class peers in academic skills. Our Follow Through achievements, however, don't show what we are actually capable of doing, because we do not have fully-implemented sites – only moderately implemented sites. The Follow Through guidelines have never permitted the kind of total system support needed to provide a full-fledged demonstration of what poor kids can achieve in grades K-3 if they receive a fully implemented, uniform, Direct Instruction approach (with trained teachers, supervisors, and directors).

The moral issue centers on this question: What does Follow Through stand for? Is it simply an experiment on human beings which has no concern for what the experiment might reveal for the millions of other "poor kids" who have serious educational needs? Is your office even remotely justified in treating all sponsors as equals and shifting (in the past two years) the emphasis from sponsors to individual sites? Or is this move designed to detract from the issues of effective approaches and make it seem that "every approach is capable or producing good results"? We would not have engaged in Follow Through for the past ten years if we had thought that there would be no attempt to weed out the inadequate model approaches, to educate both the public and the educational community about how to be effective, and to disseminate information on how to be an effective model. Furthermore, we would not have perpetuated our model if it proved to be a hoax; rather, we would have quit Follow Through with abject apology if the results had shown that our approach did not work any better than that of the typical Title I program and produced kids who performed only at the 18th percentile in reading and arithmetic by the end of the third grade.

According to your letter of October 28, 1977, "No single instructional approach, including that of Direct Instruction, was found to be consistently effective in all of the projects where it was tried and evaluated. Therefore, no overall claim of its effectiveness can be supported."

This statement flies in the face of the USOE's own Evaluation Synthesis (John Evans' office) indicating that there is one model generally effective in basic skills, cognitive skills, and on affective measures – Direct Instruction. Your statement also flies in the face of the Abt Report data – not merely in the face of data showing the relative superiority of Direct Instruction over comparison groups, but also in the face of absolute grade norm data that shows Direct Instruction to be the only approach to bring third graders at or near grade level in reading, arithmetic, spelling, and language.

Your conclusion that no single instructional approach was found to be consistently effective is spurious because: (1) Grand Rapids is included as a Direct Instruction site (for two data points); and (2) the conclusion confounds program effect variability with control group variability. Variability in program implementation must take into account absolute grade-level performance. Grade-level performance is clearly as relevant as statistical significance because if all kids were performing at grade level (on the average), there would have been no need for Follow Through. In fact, much of the rhetoric that led to Follow Through by Robert Kennedy, Commissioner Howe, Bob Egbert, and Jack Hughes, focused on the fact that poor kids perform well below grade level and are therefore preempted from "higher education" and its concomitants.

The so-called "variability" among the Direct Instruction sites is accounted for first by the unwarranted inclusion of Grand Rapids as a Direct Instruction site for two cohorts. Grand Rapids is not a Direct Instruction site. It is a Lola Davis site. You know that for most of 1972 and 1973, we did not function as a sponsor for Grand Rapids, that we did not receive funds for servicing Grand Rapids after the spring of 1973 when relationships with Grand Rapids were severed. You also know that Grand Rapids was not functioning as an implemented Direct Instruction site before the 1972-73 school year. It was not implemented – not because of our lack of effort – but because the director had different ideas about what and how to teach. You were involved in this situation and know quite well our position and the history. Grand Rapids is the only Direct Instruction K-starting site that performs relatively low in absolute performance. In fact, Grand Rapids performs an average of ½ standard deviation below the mean of other Direct Instruction K-starting sites. Examine the following summary tables that are based on data from Abt 3 and Abt 4, which show medium grade norms for Direct Instruction sites.

Although there is a variability, note that most of the variability is above the national norm (with the removal of Grand Rapids). Grand Rapids is the only site that consistently falls below national norm. Even the most casual inspection of these data suggests that Grand Rapids is "different". And you know that it is different because it has not been implemented as a Direct Instruction site.

In absolute data, Direct Instruction, even with Grand Rapids included, out-performed the other sponsors by one-fourth to one full standard deviation on MAT Total Reading, Language, Total Math, and Spelling. Furthermore, we can document the fact that what can be achieved through the Direct Instruction approach is only partly reflected in the table above. We can show, for example, that implementation was not fully achieved in any site and that we can do better. Our questions to you are: Wouldn't it behoove Follow Through to pursue the possibility that what we are saying is true? Wouldn't it be of potential value for educating disadvantaged kids to know what really could be done with optimal implementation? Wouldn't it be valuable to have a benchmark, a standard of excellence, that establishes what can be done and that can therefore serve as a goal for other projects and schools more generally?

In your letter, you assert: "Since 1968, the program emphasis . . . has been
on the cooperative development and evaluation of Follow Through projects, not
models . . ."

There are two problems with this statement. The first is that it is historically false. The second is that it places you on the horns of a serious dilemma.

1. The historical facts are these:

In his history of Follow Through prepared for the Brookings Conference on Planned Variation, Egbert asserted while referring to the planning going on in 1967: "With such limited funds, it seemed sensible to change Follow Through's primary purpose from 'service to children' to 'finding out what works'. . . . Follow Through now focused its attention on developing, examining, and refining alternative approaches to the education and development of young disadvantaged children."

Fairley's five-year plan in 1971 was aimed at disseminating successful models into Title I applications, a project that was terminated because the data were not yet in. The following quote is taken from the SRI Administrative History (p. F28, quoted from Haney, 1977, p. 36): "It was clear from Mr. Fairley's comments that his interest was largely upon overall program effects and an identification of 'best' individual sponsors to guide decisions about future Follow Through program scope." [Emphasis added.]

The following statement is found in the first page of the Abt 4 Report on the evaluation of Follow Through which has been accepted by the Office of Education and due for release to Congress shortly: "As part of this evaluation, the U.S. Office of Education commissioned Abt Associates in 1972 to

analyze the data generated by the extensive program of testing and interviewing which was part of Follow Through and to draw from them appropriate conclusions about the effectiveness of the various models' approaches to compensatory education." (Abt IV, 1977, p xxiii) [Emphasis added.]

In his history of Follow Through in Rivlin and Timpane's book on Planned Variation in Education, Elmore states: "The idea of planned variation experimentation adopted by Follow Through was, of course, very different. Instead of specifying dimensions of variation and attempting to limit the experiment to those most susceptible to definition and replication, Follow Through administrators tried to select promising, innovative program models, leaving the question of how these models differed to a later time." (Rivlin & Timpane, p. 36)

Garry McDaniels, former director of Follow Through research, makes these statements discussing the goal of the Follow Through evaluation:

"Do the various educational strategies used in Follow Through have different effects, and do the effects endure?" (Rivlin and Timpane, p. 47) . . . "The planned variation between models of various sponsors is considered to be a more reliable means of assessment than natural variation . . ." (Rivlin and Timpane, p. 48)

You assert that the emphasis of support is and has been on individual sites, not sponsors. This position, however, does not explain why Follow Through is organized as it is. If the emphasis is not on sponsor, why isn't there a "random variation" of approaches used in different sites? Why were sponsors introduced in the first place? The notion of planned variation makes very little sense and becomes self-contradictory if in fact there was no attempt to draw conclusions from a controlled pattern of implementation across sites. Furthermore, the National Office has commissioned studies to document the fact that different sponsors produce different patterns of implementation:

The Nero implementation study (1975) shows that sponsors are discriminable, that classroom features clearly identify different sponsors, and that there is a different pattern of performance across different sponsors. (What was the purpose of this investigation if there was no attempt to find out about sponsors, and their implementation across sites?)

Similarly, the Stallings and Katkowitz study, funded by USOE, showed through classroom observations that sponsors were discriminable in their implementation at different sites and at different grade levels. The study supported the conclusion that sponsorship is a viable concept and that sponsor implementation does lead to observable differences in classroom outcomes and in performance outcomes.

What other form of significant protest do we have other than quitting Follow Through, and severing association with the kind of non-relevant educational agencies that we have tried to fight during the past years?

References

Abt Associates, Education as Experimentation: A Planned Variation Model, Vol. 3, 1976, Vol 4, 1977, Cambridge, Mass. Abt Associates.

Haney, W. The Follow Through Planned Variation Experiment, Vol. 5: A Technical History of the National Follow Through Evaluation, Cambridge Mass., Huron Institute, 1977.

Nero and Associates, A Description of Follow Through Implementation Processes, Portland, OR, Nero and Associates, 1975.

Egbert, R., "Planned Variation in Follow Through," unpublished manuscript, presented to Brookings Institution's Panel on Social Experimentation. Washington, D.C.; April 1973.

Dear Sirs:

This is in response to your letter to me of November 16, 1977, in which you stated among voluminous other assertions, that a statement I had made in an earlier reply to you of October 28, 1977, was "historically false." Although, as Director of the Follow Through program, I have many constructive demands upon my time that prevent my entering into what now threatens to become an endless exchange of unproductive, widely distributed, correspondence between you and myself; I am compelled to respond to that charge of falsehood.

Since, on October 28, I was writing a letter to you and not a position paper as you have now done, I felt it unnecessary to provide annotation of the sources upon which I based my statements. I do so now, however, in view of this charge which you have so widely circulated.

The Follow Through Program Manual (draft), dated February 24, 1969, was prepared and used while this program was under the direction of Dr. Robert Egbert, whose later unpublished writings your paper quotes extensively. The program manual contains on page 2, the following statement under B. Planned Variation in a Context of Comprehensive Services.

"This is not reasonable in that it directly conflicts with the OE's decision that projects, not "models" would be presented to the JDRP. This was a carefully considered decision based on the following facts.

1. Since 1968 the program emphasis has been on the cooperative development and evaluation of comprehensive Follow Through projects not models.
   
   
2. No single instructional approach, including that of Direct Instruction, was found to be consistently effective in all of the projects where it was tried and evaluated. Therefore, no overall claim of its effectiveness can be supported."

In 1968, I was not in employ of the U.S. Office of Education. The draft program manual to which I refer, however, contained the guidelines used by the Follow Through program and its grantees to govern their operations until June 21, 1974 when, under my direction, the Follow Through program published in the Federal Register as an interim final regulation, its first official regulation (39 FR 22342 et. seq.)

I add that "quitting Follow Through" as a form of "significant protest" was your decision. In my opinion, it is a very destructive one to the Follow Through program and to the "poor kids" to whom you profess commitment.

Rosemary C. Wilson