PILOTed

How do you improve a school system?

McKinsey & Company has cataloged 525 different types of actions that school systems attempt, and most of them work, but in different situations. Many of the interventions are different depending on the stage of the schools. McKinsey breaks school systems into 4 different categories, dependent on student achievement: poor, fair, good, and great.

 

Image via Wikipedia

Poor systems routinely turn out students who do not read and write, and who cannot do rudimentary math. Their teachers are generally low skilled as well. Poor systems that are progressing to fair ones generally make progress with a top down approach, establishing relatively rigid class and lesson structures so that lower skilled educators can still achieve a minimum but uniform level of student achievement. Their goal is students achieving basic literacy and numeracy.

 

Fair systems have students who understand the rudiments of reading and math. They progress by providing more student performance information, setting pedagogy models and organizational structure, securing appropriate financing, and increasing accountability. Their goal is to get the foundations in place.

Good systems have students who are reasonably proficient and meet academic standards. These systems progress to great by establishing and reinforcing the professionalism of their educators with occupational structures and procedures similar to those of medicine or law. Their goal is to inculcate professionalism.

Great systems uniformly produce highly functioning graduates. They continue to improve by increasing local and individual responsibility, with more peer based learning for professionals and students, and encouraging innovation and experimentation.  Their theme is to  improve through peers and innovation.

The interventions for the first two levels tend to be top down, while the higher levels look to empower educators and students. As the report points out, you can prescribe adequacy, but you unleash greatness.

Six types of interventions tend to span all levels:

• Revising the curriculum
• Ensuring an appropriate reward and remuneration structure
• Building technical skills for educators
• Assessing students
• Establishing data systems
• Creating policy documents and laws

While many school systems attempt to implement these interventions, successful implementations (those that move a system from one level to a higher one) have six attributes:

1. Leaders have selected, and members of the school system follow, a critical mass of interventions that together will move the district forward. There is not just one quick fix, and it's not trying to do everything at once. Assessing the situation and then selecting a cluster of interventions that can be done well that will move the school system to the next level.
2. Leaders contextualize the interventions to the school system, generally through interactions with other stakeholders, to get the critical buy-in. Leaders have to obtain the support of stakeholders, and they do this through a combination of context, persuasion, and mandates. Leadership is a key ingredient for success, and the leaders need to balance how they communicate, what issues become matters of negotiation, and what issues are dictated.
3. The systems maintain the integrity of the interventions over time; they stick with the interventions over years so that they can achieve the desired results. Educators have to internalize the new teaching processes, changing the way they think about teaching, and this takes time.
4. There are structures that allow teachers to share practices and the system to disseminate and scale them.
5. Teachers understand and buy in to a professional career path, understanding that continuous learning, sharing, and adopting what works is part of their responsibility.
6. New leaders are cultivated, so that the leadership of the system is self-sustaining and self-reinforcing.

While the average superintendent in an urban district in the US currently stays at the position for an average of three years, the report points out that this is a formula for failure. This combination of six success factors requires change from quite a few stakeholders, many of whom will normally strongly resist change. All the systems that have overcome that inertia have initiated change with the introduction of a highly energized competent leader who stays for an average of at least six years, but that is generally not sufficient. Most also have experienced either some political or economic crisis or a publicized analysis or report that is highly critical of the system.

 

Image via Wikipedia

Hope for improvements in our school systems thus lie with some external event triggering a desire for change in the community or school system; the ascendency of a good leader; following the six attributes of success; and being conscious of the school system's level.

 

Somehow, that doesn't sound so complicated.

Related articles

• Transforming the Public Education System Through Data (impact.webershandwick.com)
• Editorial: There Has to Be a Better Way to Grade Tests (nytimes.com)

My Problems with the Common Assessment

The PARCC (Partnership for the Assessment of Readiness for College and Careers) released a draft of their assessment frameworks for ELA and Math, and asked for feedback: http://www.parcconline.org/parcc-content-frameworks.

Here are my three major objections.

The Industrial Age Paradigm

The assessment framework reinforces the current model of schools as factories and children as raw material to be forged into a uniform product. The assessment framework is still based on a cohort of students starting in Kindergarten and learning a prescribed dose of skills and knowledge each year through 12^th grade. For example, while the ELA standards allow teachers or schools to select the specific reading passages, passage specifications are bounded by grade:

Use of grade band-level complex text: Leveled texts that are below grade band level in complexity are not a substitute; the standards indicate students should be reading grade band-level complex text. Flexibility is built in for educators to build progressions of more complex text within grade band levels (e.g., grades 4–5, 6–8, 9–12) that overlap to a limited degree with earlier bands, but reading text from the appropriate band level lies at the core of the Model Content Frameworks.

Students are children and children are individuals as are teachers. Why can't we bust out of the early 20^th century mentality that everyone should be learning the same things in the same years?

The flawed concept of effective valid standardized tests of cognitive behaviors

There are basically four types or results one can assess: feelings, knowledge, behaviors, and results.

• Feeling assessments involve questions like, "how confident do you feel that you learned how to do X?" Feelings are the least reliable and least expensive way to assess.
• Knowledge assessments are more objective. Multiple choice questions "Which is the main idea in the following paragraph?" are one way of testing knowledge, as are short answer questions, and, in some cases essays can also test knowledge (or skills). Knowledge assessments test whether the learners can recall facts or procedures, they are fairly straightforward to develop and score, although more expensive than feelings assessments. One reason knowledge assessments are flawed is that they can only test a subset of what a student has learned over a period of time. For example if a teacher happens to have spent time on the specific areas tested (from, say, tailoring classes to what was tested in previous years), while neglecting areas that aren't being tested, the test will give a false result on total knowledge. Most current high stakes assessments test basic knowledge.
• Behavior assessment tests whether the learner applies knowledge or skills when placed in a situation that calls for them. These assessments are much more expensive to develop, and need to place students in novel situations, so they cannot merely regurgitate facts or directly apply algorithms they have memorized. When I was developing content and assessments, our rule of thumb was that, when teaching and then using behavior assessment, we needed to allocate 1/3 of the budget for the cost of assessment development and administration, and content development and instruction was 2/3.
• Results assessment tests whether the students are better able to achieve organization-wide goals. In the case of education, we'd want to evaluate whether they become better citizens, better workers, or better parents. This type of assessment is extremely difficult; can we even come to an agreement on what a better citizen, worker, or parent is? How could we possibly measure it? How could we determine how student activities in 1^st grade make students better citizens, and what could we do with that information?

The goals of the PARCC Assessment Framework (assessing to know if students have the cognitive skills and knowledge to perform real tasks, and determining interventions when they do not) are noble. But my experience tells me that the cost trade-off of deep assessment versus saving money never works out. Following my old rule of thumb, where 1/3 of the cost is assessment, and 2/3 of the cost is teaching, you could reallocate that money and effort to essentially teach 40-50% more by using a simpler assessment. Would you rather teach more students more skills and knowledge (and not completely know the effectiveness) or teach less, but assess better? When the final dollar allocation decisions get made, we always seem to tip the scales toward reducing assessment costs in favor of teaching or savings.

Thus, to me, it seems more likely that PARCC will eventually fall down when it comes time for states to put money behind developing and scoring the assessments, and we will end up with tests not significantly better than the ones we currently use.

High stakes assessments always get gamed

If we are going to use assessment to determine teacher pay, teacher retention, and funding for schools and districts, the people being rewarded or penalized will search for ways to maximize their outcomes, and some of those will affect the validity of the students' test results. I know I am not the only person who writes about this, but here are three previous PilotEd articles that address this issue:

Interview with Sharon Nichols

Mapping state proficiency standards to the NAEP

Many states cheat on standardized tests

What's my recommendation?

With the current direction of the Common Assessment, it's as if we've taken some of very smart, competent people, and we've told them, "Build a great road to that mountain over there." And they'll spend a lot of time, effort, and money building a great road to the mountain. But what we really need is a road to the sea, so what we'll get is a great road leading us to a place we don't really want to go to.

The whole debate about what students should know, when they should know it, and how we assess it is extremely important for our society. Rather than using that debate to frame high stakes assessments that merely tweak our current educational paradigm, we should

1. Rethink what it means to offer education.
2. Use the Common Core Standards and the PARCC frameworks as the bases for defining the skills and procedures needed by educators and parents.
3. Build out corresponding community, pre-service, and in-service curriculum for all who will guide student learning.
4. Develop resources for educators and parents to unleash to potential for each child.

Related articles

• Who Wrote the Common Core Reading Standards? (dconrad3.wordpress.com)
• Elimination of evaluation/assessment practices? (education420.wordpress.com)
• Module Six - Standards Based Assessment (tracyfeltonbportfolio.wordpress.com)

Education Questionaire for "States Cheat.." Blog Post

The embedded questionaire did not print in the emailed version of the Many States Cheat on Student Proficiency blog post; it does appear on the web version of the blog.

Here is a direct link to that survey:
https://docs.google.com/spreadsheet/viewform?formkey=dHZYSDBiNUJzbnBNTUE2ajdJMndteEE6MQ

Please feel free to pass this on, and we will summarize the results on August 22.

Many States Cheat on Student Proficiency

Virtually all states have proficiency standards below those of the National Assessment of Educational Progress.

There is a wide variation in what states call proficiency; students who achieved proficiency in the lowest 5 states would have been labeled basic in the highest 5 states.

There are 40 cases of states reducing the rigor of their state high stakes tests between 2005 and 2009.

In over half the cases, self-reported (by the states) improvements in student performance do not correlate with student NAEP test score results.

This is from the NAEP report Mapping State Proficiency Standards onto the NAEP Scales at http://nces.ed.gov/nationsreportcard/pubs/studies/2011458.asp.

This isn't a new phenomenon. We knew this from the previous report in 2009.

So what should we do about it? Should we make the NAEP tests standard across the states? Would we be better off just making sure everyone knew how their state student achievement reporting compared to national averages and let changes occur from local pressure? Should we abandon high stakes testing? Should we try to get testing "right" and then set national standards?

Please let us know what you think by responding on the following form. We'll collate the results in a post on August 22.

Related articles

• Title I Schools Combat Low NAEP Scores with Imagine Learning English (prweb.com)
• Standardized Tests' Measures Of Student Performance Vary Widely: Study (huffingtonpost.com)