Trouble with Data: Don't be Fooled by the Numbers
Data seems solid, substantial, productive, professional.
And, there is so much of it.
Sources of data are everywhere, and teachers are exhorted to use data in making instructional decisions and educated choices. No "true professional" would leave problem-solving and decision-making to intuition, creative serendipity, hunch, or chance when they could engage the superior qualities of their mind in goal-directed, data-driven executive-like skills.
But, there are leaks in the dike that solid data is supposed to provide, and the legs of the data step-stool are a bit wobbly…the underpinnings of data could crumble or collapse at any time.
Data would seem to fit "hand and glove" with the Science of Teaching; but, instead of a mortise and talon relationship, data and teaching are often "strange bedfellows."
Problems with Data
What are the problems with data if data is so scientific and professional?
- Data accumulates fast. It floods our minds and requires time and resources to collect and manage
- So much of the data that we collect is raw, uninterrupted. By itself, data is without meaning
- Data take time to collect, and even more time to separate, sort, and make sense out of it
- Data can be trivial, but can look like it is meaningful. For example, grades and test scores
Grades provide "constantly generating" data that has little meaning without the interpretation of the teacher that created them.
For example, does that "D" in English mean that the student…
- Nearly failed the big test
- Was absent too many time and missed a couple of quizzes
- Didn't turn in his or her homework. Hint: The dog ate it!
- Didn't complete the research report on time and lost 20 points
- Just moved from Mexico and doesn't speak English well, yet
- Etc. Who know but the teacher?
Test scores can also be misleading.
For example, the scores on the high stakes test can all be 71 in one class and 100% of the teacher's students passed. In another class, the scores can all be 69 and 0% of the students passed. These teachers would look to be miles apart in their success, yet, their scores are statistically "dead-even."
In another test score example, the average classroom scores for one teacher are 4.7, exactly at grade level. However, another teacher has a class average of 4.8. When these scores are reported to the local newspaper and published by teacher name, community members come to believe that the second teacher with a "0.1" higher class average is a better teacher. But, nothing of the sort. These scores also indicate a "dead heat."
A Little Data is a Dangerous Thing
Limited understanding of the conditions and environment where data were collected, and meager control upon how the data were collected can result in unwarranted assumptions. Here are some pitfalls:
- Comparing "square pegs" with "round pegs"
- Comparing unstructured data to structured data
- Comparing data related to "fuzzy" categories
- Comparing data where subjects reside in multiple categories
- Comparing minor differences because the numbers
- Finding "false positives" and rejecting "false negatives"
- Bogging down in data collecting and getting sloppy in data acquisition because it is too much work
- Becoming stressed and overwhelmed by too much data, i.e., information overload
- Collecting data without a clear plan for how that data will be used
- Creating reports and presentations based on correct data but false assumptions
- Missing the intervening variable
- Identifying the wrong independent variable (The one you can change)
- Measuring the effect of items that are beyond your control
- Etc. ad infinitum
"Grain of Salt" Data Management Plan
The "Grain of Salt" Data Management Plan can also be called the "Information Overload Survival Plan."
Here are the rules to follow:
<
- Only collect data after you know what you are going to do with it
- Make sure that what you are going to do with the data has been proven to achieve measurable results
- Collect the least amount of data
- Collect data in the easiest possible way
- When data seems to defy common sense, go with common sense every time
- Don't believe anything that software sales people tell your district's administrators and decision-makers about the ability of their product to increase test scores
A Matter of Trust
When it comes to where to place your trust; trust your hunches, intuition, "gut feeling" over numbers and averages.
Test every assumption that is inferred by the data before committing to major changes in a system that is already working.
Sidebar
Classroom Toolkit reported on your best data management resource in a previous issue, A Teacher's "Bionic Brain:" The World's Best Super Computer
Knowing your students as people first, dynamic personalities, feeling and thinking beings…is more important than analyzing a test-score print out.
And, communicating and relating to your students will bring them farther along in life"as well as give them the highest edge on test score performance, too.
Use data if you must, but first, be a great teacher.
Like technology, data is only as good a tool as the teacher who knows how to use it.
Use data wisely. Or, don't use data at all.
