#Big Picture
During the last five daily blog posts, I have worked my way through the five rules from P. David Pearson and the #ILA19 panel session at 7 a.m. Saturday titled: “What Research Says About Teaching Reading and Why that Still Matters.”
Rule 2 #Research Applied Evenly
Rule 4 #Full portfolio of methods
Rule 5 #Evidence, not a strawperson
Understanding the research in today’s world takes some work, some thinking, and a good hard look at the evidence, the word that appears in both rule 3 and rule 5.
A week ago, this was how I started my first draft for the series. I quickly discovered as I wrote that this look at the Big Picture was the ending of the series instead of the beginning. The REAL beginning was the panel presentation that recentered some beliefs in processes and brought back a review process used by our Statewide Literacy Team in the past.
So let’s get started. “It was a dark and stormy night.” (I love how Snoopy works that into every story!)
Compare these headlines:
- ‘No Progress’ Seen in Reading or Math on Nation’s Report Card
- Screen Time Up as Reading Scores Drop. Is There a Link?
- The One And Only Lesson To Be Learned From NAEP Scores
-
Results are in: Mississippi Students #1 in the Country for Reading Gains
- Mississippi: Miracle or Mirage – 2019 NAEP Reading Scores Prompt Questions Not Answers
Match the quotes with the titles above. 1. #Headlines
_____ NAEP is extraordinarily clear that folks should not try to suggest a causal relationship between scores and anything else. Everyone ignores that advice, but NAEP clearly acknowledges that there are too many factors at play here to focus on any single one.
_____ In reading, Mississippi was the only state to improve in 2019 in 4th grade and Washington, D.C. (considered as a state) was the only one to improve in 8th grade. (The District of Columbia, in fact, showed the fastest gains this year of any state or large school district.)
_____ Todd Collins has raised another important caveat to the 4th-grade reading gains in Mississippi because the state has the highest 3rd-grade retention percentages in the country. . .
_____ Mississippi was the only state in the country to improve reading scores, and was number one in the country for gains in fourth-grade reading and math, according to newly released test results.
_____ Students have actually lost ground since 2017 on both of the NAEP’s main reading content areas: literary experience, such as fiction analysis, and reading for information, such as finding evidence to support an argument. Both grades declined significantly in both areas from 2017 to 2019, but the drop was larger for literary skills.
Which ones seemed pretty obvious?
Which ones took a bit more thought?
And then which two came from the same publisher?
. . .
. . .
#1 Headlines and text that supports or matches the headline.
Answers:
3, 1, 5, 4, 2.
Same Publisher: 1 and 2 were both EdWeek
Of the five articles, where would you expect to see research?
Tip: #2 showed that data was reported but not research in article #4.
What is the best evidence?
When I return to “Results are in: Mississippi Students #1 in the Country for Reading Gains,” I actually have more questions after more reading. Especially after reading this article: “Here’s What All the NAEP coverage missed.”
What if the reading gains are the result of higher beginning points every year?
2.#Research Applied Evenly
What would be worthy of studying?
- Is the gain the result of instruction delivered to the students?
- Is the gain the result of the professional development provided for the teachers since 2013?
- Is the gain the result of the addition of coaches in the lowest buildings (in the fall of 2018)?
- Is the gain the result of the retention policy?
And that takes me back to Paul Thomas’s blog (#5 above). And this updated section:
- UPDATE: Todd Collins has raised another important caveat to the 4th-grade reading gains in Mississippi because the state has the highest 3rd-grade retention percentages in the country:
But Mississippi has taken the concept further than others, with a retention rate higher than any other state. In 2018–19, according to state department of education reports, 8 percent of all Mississippi K–3 students were held back (up from 6.6 percent the prior year). This implies that over the four grades, as many as 32 percent of all Mississippi students are held back; a more reasonable estimate is closer to 20 to 25 percent, allowing for some to be held back twice. (Mississippi’s Department of Education does not report how many students are retained more than once.)
This last concern means that significant numbers of students in states with 3rd-grade retention based on reading achievement and test scores are biologically 5th-graders being held to 4th-grade proficiency levels. Grade retention is not only correlated with many negative outcomes (dropping out, for example), but also likely associated with “false positives” on testing; as well, most states seeing bumps in 4th-grade test scores also show that those gains disappear by middle and high school.
After several questions about “retention” and/or “intervention” and/or “multiple attempts on the state assessment,” maybe this is a focus for research. What data do we have? What data do we need to collect? What other questions bubble up?
- Did students who did not meet the proficiency level have higher absenteeism that proficient students?
- Did any specific classrooms have higher growth than others?
- What do we know about the implementation of the teacher training?
This “study” may require some additional data collection but it could be undertaken relatively quickly to form some general ideas yet this year.
Because I want to reduce the need for intervention, I might also explore this chapter from Regie Routman’s, Read, Write, Lead. (Link) 3.#Best Evidence 4.#Full portfolio of methodology
What I wouldn’t do is:
Give the 4 point credit to ANY of the above areas without study.
Blame teachers for not implementing “enough” or “correctly” without study.
Say that Mississippi has a program that should be replicated in every state because we don’t know the amount of resources that it took to get these results that are not sustained through 8th grade . . . without study. 5. Evidence, not a straw person.
The purpose of this post was to pull together a topic currently in the literacy field, generate some questions, look at the data, and apply the 5 rules from the Research presentation. In less than an hour my questions were generated and this post was written. A beginning application. A beginning look at the Big Picture.
You can do this.
You must do this.
You need to verify the accuracy of what you are reading. Find a partner and get started!
#Research Applied Evenly
#Headlines dealt with “Rule #1.” (Link) P. David Pearson at #ILA19 was a panel member for a Saturday 7 a.m. session titled: “What Research Says About Teaching Reading and Why that Still Matters.” Dr. Pearson proposed several rules for our work and I have been considering this second rule for the last few weeks as I have read across Twitter, blogs, emails, newspapers and journal articles.
Rule 1: Policymakers have to read beyond the headlines.
Rule 2 is captured here.
Let’s return to
Results are in: Mississippi students No. 1 in the country for reading gains (Link)
What research is reported?
All the research?
Dictionary.com defines research as:
Go check out the article and identify the “research” you find.
. . .
Hmm
. . .
Hmm
. . .
Data
Data
Data
Reporting of “findings” or “results”
Hinted at in this section:
The Mississippi Department of Education attributed the some of the continued success in reading scores to the Literacy-Based Promotion Act, a law that went into effect in 2013 that requires third-graders to pass a reading test before they can be promoted to the fourth grade. This year marks the first where students had to hit a higher bar to move up a grade.
“Mississippi has entered a new era of public education,” said Jason Dean, chair of the Mississippi State Board of Education. “Our significant improvements in teaching and learning have made Mississippi a national leader for improving student success in education.”
The linked article about “a higher bar” took me to this article with this quote in the final paragraph.
Woods is one of dozens of literacy coaches working in classrooms across the state as thousands of third-grade students prepare for their final chance at passing a critical reading exam. Should they fail, the possibility looms heavy that they’ll have to repeat the grade. (Source Link)
Facts
- Picture dated 6/17/19 labeled with coach and students
- Dozens of literacy coaches
- Thousands of third-grade students
- PREPARE
- Final chance at passing a critical reading exam
- possibility to repeat a grade
Questions/ Wonderings
- Many schools in MS, begin in the second week of August. Was this a summer school program extending the year?
- Final chance: How many opportunities had students already had for this test? Beginning when? How frequently could students retake for another chance?
- What is this test?
- Is this test aligned with NAEP?
- What is the technical adequacy of this test?
- Have third grade teachers in MS seen the test questions?
- Does test prep occur during the school year in the third grade classrooms?
- How many third grade students had to repeat a grade?
- What are the “significant improvements in teaching and learning”? (Jason Dean)
- …. (Add your own)
Additional information gathering – Mississippi Dept. of Ed. 2013 Literacy-Based Promotion Act (link)
- Train K-3 teachers, curriculum specialists and other educators
- Research-based instructional strategies
- 2014-15 retention for lowest students unless “good cause exemption”
- Law modified in 2016
- Fall of 2018 literacy coaches were deployed
Additional “digging” on the site, training was in LETRS (subject of IES study link)
Result from RTC study: What did the study authors report about the efficacy of LETRS?
Providing second-grade teachers training based on the LETRS curriculum (with or without the instructional coaches) increased their knowledge of reading instruction techniques and their use of explicit instruction. However, it did not increase the reading test scores of their students {emphasis added}.
The authors estimated effect sizes on reading scores that ranged from 0.03 to 0.08. These estimates were not statistically significant.
Questions about the training:
- How many days of training did the Mississippi teachers have?
- What was the implementation plan for reviewing the instruction in the classrooms?
- What percentage of teachers implemented their training as presented?
- What percentage of teachers were observed for fidelity of implementation?
- What percentage of teachers studied their own implementation of the instructional changes?
- Mathematically, what was the benefit to students in terms of Cost of Teacher Training (K-3) x # of Training Days (cumulative # for all years) / Number of Students (counted only once)?
What do I now know?
So some facts were reported in the initial article.
Some generalizations about student performance were made.
No research was reported. In fact a prominent journalist claims “There’s no way to know for sure what causes increases in test scores.“
I added in research from a What Works Clearinghouse report on the effectiveness of LETRS.
How can causation or correlation be implied for this “growth of 4 points”?
And what is the significance of “outlier growth” in typical research?
Studying the “growth” for 4th grade students in MS would be an appropriate action from a group who advocates for “science”.
Rule 2 for RESEARCH was not applied in the original article. You can “judge” whether any research is applied in additional articles on the same topic.
CHALLENGE:
Choose an article, any article, that supports “Science of Reading”. Identify the precise research in the article. Study it. What do you really find?
Mississippi 2019 NAEP Results Examined – Link
#Headlines
I remember professional learning with Emily Calhoun where we discussed how the “title” of any book was the promise the author makes to the reader about what the book will be about. A book is bigger than an article in a newspaper or a magazine. Is a headline similar? This lead me to some research about headlines and the types, functions, and even the definitions. Scacco and Muddiman in “The Current State of News Headlines” report four functions of headlines.
The news headline can serve a variety of functions, including story summarization, interest generation, immediacy satisfaction, and attention direction. (Link)
-
- Story summarization
- Interest generation
- Immediacy satisfaction
- Attention direction
Consider this headline. Which function fits?
Results are in: Mississippi students No. 1 in the country for reading gains
Do you know enough to make a decision?
This headline was published in Mississippi Today and according to its website,
Mississippi Today is a nonprofit 501(c)(3) news and media company with a forward-facing mission of civic engagement and public dialog through service journalism, live events and digital outreach. (Link)
Does that descriptor of the publication change your mind about the function of this particular headline?
The first paragraph of this article says,
Mississippi was the only state in the country to improve reading scores, and was number one in the country for gains in fourth-grade reading and math, according to newly released test results.
and yes, it was published the day after the NAEP results were released. (Link)
Has your view of the FUNCTION changed based on a) additional knowledge about the publisher; b) the knowledge of date of publication; and/or c) the first paragraph of the publication?
Which best fits your thinking?
-
- Story summarization
- Interest generation
- Immediacy satisfaction
- Attention direction
Why does it matter?
P. David Pearson at #ILA19 was a panel member for a Saturday 7 a.m. session titled: “What Research Says About Teaching Reading and Why that Still Matters.” Dr. Pearson proposed several rules for our work and I have been considering this first rule over the last two months as I have read across Twitter, blogs and newspapers.
Rule 1: Policymakers have to read beyond the headlines.
I would be remiss if I didn’t note that teachers, administrators, college instructors, parents, and anyone in the U.S. REALLY needs to read beyond the headlines. But careful attention is required particularly in the field of reading, reading instruction, and any “claims” in the headlines about reading pedagogy.
What does this article tell us?
Gains in 4th grade reading in Mississippi.
Only state with gains in 4th reading.
No gains in 8th grade reading in Mississippi.
The gain was 4 points.
“The 2019 results mark the first time Mississippi has met or outperformed national averages.” (In 1992, Mississippi was 16 points below the national average.)
Mississippi scores declined from 2009 to 2013.
And the “credit” for the “increase in scores”:
The Mississippi Department of Education attributed the some of the continued success in reading scores to the Literacy-Based Promotion Act, a law that went into effect in 2013 that requires third-graders to pass a reading test before they can be promoted to the fourth grade.
Do those facts match up with the function of the headline?
For additional practice let’s consider a second view of the Mississippi scores found in this blog post from Paul Thomas last week.
Mississippi Miracle or Mirage?:
2019 NAEP Reading Scores Prompt Questions, Not Answers
Which function matches this headline?
-
- Story summarization
- Interest generation
- Immediacy satisfaction
- Attention direction
And here’s the first paragraph of the blog post.
There is a disturbing contradiction in the predicted jubilant response to Mississippi’s outlier 4th-grade results from the 2019 NAEP reading test. That contradiction can be found in a new article by Emily Hanford, using Mississippi to recycle her brand, a call for the “science of reading.”
What do you believe is the purpose of this headline? Is it similar to the previous article? Or different?
Dr. Thomas then quotes two paragraphs from E Hanford’s own post:
The state’s performance in reading was especially notable. Mississippi was the only state in the nation to post significant gains on the fourth-grade reading test. Fourth graders in Mississippi are now on par with the national average, reading as well or better than pupils in California, Texas, Michigan and 18 other states.
What’s up in Mississippi? There’s no way to know for sure what causes increases in test scores [emphasis added], but Mississippi has been doing something notable: making sure all of its teachers understand the science of reading.
Paul answers Hanford’s claim that “there’s no way to know for sure” with
To be fair, there is a way to know, and that would be conducting scientific research that teases out the factors that can be identified as causing the test score changes in the state.
Scientific research . . .
A broader look at the data suggests that in 1998, Mississippi was only 10 points below the national average. What changed to cause growth between 2005 -2009?
Blog post Source Link
Facts/Questions from this article:
Is the 2013 legislation responsible for any growth? What research supports this hypothesis/generalization?
Is this the role of NAEP data? Should it REALLY be?
What about the 8-point jump in 4th-grade reading in MS from 2002 to 2009 with no explanation?
Original author Hanford used correlation (not scientific) instead of causation (scientific).
Premature?
Irresponsible?
No research?
No evidence?
In any informational text, the information that is included is always critical. But equally important (Hat tip to Katie Clements) is the information that is left out. And the questions that remain after reading the articles. One place for readers to begin in with the promise of the headline, the match between the headline and the article content, and the basic functions of a headline are one entry point.
Did the articles match up to the “hype” of the headlines?
Did they serve the function?
Why is P. David Pearson’s rule about headlines important?
Skinner, K. Results are in: Mississippi students No. 1 in the country for reading gains. Retrieved from https://mississippitoday.org/2019/10/30/results-are-in-mississippi-students-no-1-in-the-country-for-reading-gains/ on December 7, 2019.
Thomas, PL. Mississippi Miracle or Mirage?: 2019 NAEP Reading Scores Prompt Questions, Not Answers. Retrieved from https://radicalscholarship.wordpress.com/2019/12/06/mississippi-miracle-or-mirage-2019-naep-reading-scores-prompt-questions-not-answers/ on December 7, 2019.
Resource - Full 