Litmus Test for Urban
by Siegfried Engelmann
July 2005
A man who had never seen a
bicycle received a kit, completely unassembled, with instructions. The man felt
he had a sense of machinery, and he didn't see the need for some parts. Also,
he added parts he felt were necessary. The man believes that his assembly
process was appropriate. His bike runs, but it's hard to pedal and hard to
steer. Question: How much of the observed performance is created by the way he
assembled the bike? If the man had accurate information on bikes that had been
assembled properly, he would have a basis for comparing his bike with the
standard model. As it is, he doesn't know what the bike's potential is because
the bike's performance is the product of two variables, the machinery and the
way it is assembled.
The same problem exists with
urban school districts. They implement approaches according to their rules or
standards, rather than the developers' guidelines. The result is the same as
that of the bike. The performance of students is now the product of two
variables—the approach and the way the district implemented it. Just as the man
has no basis for comparing his bike with those assembled according to the
manufacturer's specifications, the district has no basis for comparing the
results it achieved with those that would be generated by the developers'
guidelines.
The district could obtain
this information easily, however, simply by implementing the approach in a few
schools according to the developer's guidelines. That's what the litmus test
is—a controlled, carefully monitored, small-scale test of possibly effective
approaches. Unfortunately, there is nothing to suggest that urban districts are
capable of implementing any effective approach with fidelity. That may be why
they haven't discovered what works well.
It's important to find out
if urban districts can pass the litmus test because if they can't, they should
be reconstituted so they have the capacity to implement with fidelity, and so
they base their decisions on outcomes of small-scale tests of well-implemented
approaches.
For instance,
Apparently
Even though the Direct
Instruction implementation in
Maybe
Or possibly, small schools
for at-risk populations are extremely hard to implement. For City Springs, its
small size created a host of implementation problems. If a couple of teachers
were absent, it was very difficult to cover for them with trained teachers or
aides, much harder than it is in a larger school like Carver (which has many
more classrooms on each grade level).
Also, in a well-implemented
small school, grouping students homogeneously for instruction becomes a
nightmare in grades 4 and above. The reason is that a large percentage of
incoming students perform lower than any continuing students in these grades.
Accommodating new entrants is far more challenging than it is in a larger
school, which may have more than one classroom per grade dedicated to
accommodate incoming low performers. A small school with more than 25% annual
turnover (like City Springs) often has to penalize the continuing students by
slowing their performance so the classroom is able to accommodate
low-performing incoming students.
Other small-school problems
include training and deploying coaches, accommodating students who have been
absent for a while, and training teachers who are performing unacceptably.
Possibly
But even if
"Pathways" has the potential to work well and the management has
super smart strategies for implementation and training, what evidence is there
that the district could implement the approach in a way that would achieve its
potential? The answer is revealed by the litmus test. If districts don't have
the machinery needed to implement on a small scale, there is no reason to
believe that they'll be able to faithfully implement anything that is
instructionally sound on a larger scale.
The format for the litmus
test is a parallel to how the man with the bike could secure information about
the bike's potential. Instead of ordering one bike, the man orders two,
assembles one according to the book and one his way. Now he has a strict basis for
comparison and is able to evaluate which practice is best. For the litmus test,
the district identifies four models. Two would have substantial data of
effectiveness (such as DI) and would be implemented the developers' way. Two
would be approaches that the district prefers (such as the Renaissance Schools)
and would be implemented the district's way. The performance of the effective
models would serve as the yardstick for evaluating the other two models.
Each approach would be
implemented for three years in three comparable at-risk schools (12 schools
total). The district would necessarily waive whatever policies and practices
interfere with the effective models being implemented according to the
developers' specifications (given that these are legal, humane, and feasible).
The district's standards, preservice and inservice training, and other practices may be in conflict
with the models being tested. Waiving standards and procedures should not be a
serious barrier. After all, the schools failed even though they followed these
standards and practices.
Certainly, the effective
models would have to stay within reasonable budget limitations and could not
demand things like unusually gifted teachers or three aides in every classroom.
A reasonable demand, however, would be that the principals
are to be directed to follow the model's provisions. (This did not
happen with DI in
The evaluation of the
district's performance on the litmus test would be based on both the degree to
which each model was implemented, and how well the schools performed. The
evaluation of implementation fidelity would be performed by an independent
agency and use objective measures—how well teachers and principals follow
specified procedures and schedules. Students would also be tested on
standardized achievement tests and state tests (with the testing being
scrupulously monitored) but the results would not be released to the school or
the administration until the three-year litmus test was completed. This
provision assures that judgments of how well a school is implemented would not
be biased by how well students performed on these tests.
The litmus test would
provide at least five benefits for the district.
1. It would generate
accurate comparative data about what works well, and about the relative cost of
various approaches.
2. It would reveal
modifications in the district's infrastructure that are needed to empower the
district with the capacity to implement approaches faithfully.
3. It would save millions of
dollars on large-scale implementations of approaches that would not produce
worthwhile performance gains in a small-scale test.
4. It would protect large
numbers of students from being subjected to shoddy instruction by limiting the
number of students used in the "experimental" test of new approaches.
5. It would establish a
basic professional standard for the district, which is that nothing is adopted
until it demonstrates its worth in a small-scale, carefully monitored study.
The litmus test is not only
the scientific or logical way for districts to discover how effective various
approaches are and what's wrong with the district practices. The litmus test is
also what any smart business would do—make prototypes and test them rather than
launch into full-scale production without having any solid performance
information about the product or how to use it effectively.
Ironically, some urban school
districts claim that they fashion themselves after hard-nosed business practices.
They have CEOs instead of superintendents, and their rationale for doing things
makes reference to business. For instance,
| ©2005 Siegfried Engelmann |
Litmus Test for Urban School Districts
|