Homework Research

In my last post, I speculated that there were three reasons I read educational research.

1. I encounter it (via Twitter, blogs, or in journals) and I’m curious, so I read it.
2. I deliberately seek it out to confirm a bias. (Don’t judge me. We all do this.)
3. I’m genuinely interested in what the research has to say on a certain topic, so I search for it.

Since biases are fun, let’s look at an article I dug up for the second reason. I’ve made my views on homework pretty clear on this blog in a couple of posts. Here’s a study I found on the subject of homework. Unfortunately, it failed to confirm my bias.

Are We Wasting Our Children’s Time By Giving Them More Homework?

The study is by Daniel J. Henderson, of New York and was published as IZA Discussion Paper No. 5547, March 2011.

The abstract reads:

Following an identification strategy that allows us to largely eliminate unobserved student and teacher traits, we examine the effect of homework on math, science, English and history test scores for eighth grade students in the United States. Noting that failure to control for these effects yields selection biases on the estimated effect of homework, we find that math homework has a large and statistically meaningful effect on math test scores throughout our sample. However, additional homework in science, English and history are shown to have little to no impact on their respective test scores.

Yikes. Math homework has a large and statistically meaningful effect on math test scores throughout our sample? Uh. Oh. I guess I’d better read more than just the abstract and see if I can figure our what is going on. The math used in the study is complicated. That might make it tricky to read.

Here’s something I wonder about. Page 9.

…higher able students benefit more from additional homework.

Perhaps higher able students are the only ones who actually do the homework, because they’re the only ones who are capable of doing it.

Later, on page 17.

Taking the Peabody Individual Achievement Test in math as our benchmark, the gain from math homework (1.77 points) corresponds to one-fourth of the raw black-white test score gap between the ages of 6 and 13

My question would be: Can we be sure the gain on that test is solely attributable to homework? Maybe we can. I’ll admit to not fully understanding the tables in the study.

Here’s a finding on page 19 that I am glad to hear. At least one of my biases was confirmed by this study.

The teachers Treatment of the homework (whether it is being recorded and/or graded) does not appear to affect the returns to math homework.

 

Educational Research

I’m going to write some posts over the next little while about educational research. Just before Christmas, Michael Pershan and Chris Robinson were going back and forth on Twitter about research vs. blog posts.

“If teachers can rely on blog posts, where does that leave ed research?”

That question got me thinking a whole lot about how and why I use educational research and how and why I use blog posts.

I read research for several reasons.

1. I encounter it (via Twitter, blogs, or in journals) and I’m curious, so I read it.
2. I deliberately seek it out to confirm a bias. (Don’t judge me. We all do this.)
3. I’m genuinely interested in what the research has to say on a certain topic, so I search for it.

My recent blog post about delayed feedback falls into the first category. A colleague showed it to me and I was curious, so I read it.

I tend to mine blogs for ideas that I can use immediately in classrooms and workshops. Those ideas don’t have to be researched based, in my opinion. The fact that a colleague tried it already and it worked for her is sufficient for me to try it out. That endorsement is worth one class period or one unit of study of my time. I see these shared ideas the same way I saw lunchroom conversations in the 1990s. “I did this cool thing in my class today. You should try it out.”

If I were contemplating a major shift in my practice, I’d probably go to research in addition to listening to colleagues. SBG would be an example of something I’d research before changing my whole practice. A blog might inspire me to try it, and the research would confirm that it’s worth doing. One year in Math 8, I did the entire course in cooperative learning groups and activities. That’s a big commitment. That’s a big shift. Research supported and justified my change.

In the next few blog posts, I’m going to look at some of the research I’ve read over the past few years. I’ll explain how I happened across it, and how I use it now.

 

Delayed Feedback

It’s been an interesting enough week in the assessment world that I’m compelled to blog for the first time in a long time.

Early last week, I encountered this “Focus on Formative Feedback” literature review by Valerie Shute.

Table 4, near the end, on page 179 lists “formative feedback guidelines in relation to timing issues.” Shute recommends using immediate feedback for difficult tasks, and delayed feedback for simple tasks. She says that to promote transfer of learning, teachers should consider using delayed feedback. To support that claim, she says,

According to some researchers (e.g., Kulhavy et al., 1985; Schroth, 1992), delayed may be better than immediate feedback for transfer task performance, although initial learning time may be depressed. This needs more research.

Then, just yesterday, Dan Meyer jumps in with a post on delayed feedback.

My gut says that the timing of the feedback is far less important than the quality of the feedback. Dylan Wiliam has entire chapters dedicated to providing feedback that moves learners forward. Next steps are useful to all students. Evaluative feedback that evokes emotion isn’t particularly useful to anyone.

I’m not sure this does need more research.

The Sky Is Falling!

There’s been a lot of twitter and media buzz about a new app that scans math questions and gives answers.

Dan Meyer has compiled some thoughts on the app over on his blog, and he has been commenting on Twitter as well.

 

I decided to test Dan’s comment with (what else?) a test. I gave the app one exam from each of grades 7, 8, 9, 10, 11 and 12. My conclusion is that it doesn’t solve them anywhere near as well as most kids would.

Full disclosure: The grades 10, 11 and 12 exams were ones I created, and I’m always conscious of trying to avoid having questions that can be answered with a calculator alone. The grades 7, 8 and 9 exams were from a publisher.

I tried to pick topics it had a shot at solving. I tried to pick topics with mostly number and equations.

Grade 10 – Algebra and Number

The app got 0/30 on my exam. On the questions I thought it should be able to answer, it got 0/10.

This one was its most blatant error. I did have it centered properly prior to snapping a screenshot. It registered the 2, and ignored it.

 

These were its first steps. It had trouble recognizing that square bracket.

Grade 11 – Radicals and Absolute Value

The app got 3/32 on my exam. On the questions I thought it should be able to answer, it got 3/13. I was a little surprised. Clearly I need to tweak some questions. Here’s one it got right.

 

Grade 12 – Exponents and Logarithms

The app can’t recognize logs, or manipulate anything but the most rudimentary equations. It got 0/30 overall and 0/9 on the ones I thought it should get.

Math 7 – Integers

The app struggles with brackets. I hovered over expressions like (-2) + (4) – (-7) endlessly waiting for an answer of any kind (right or wrong) and never got anything. It got 0/20 overall and 0/9 on the ones I thought it would get.

Math 8 – Fractions

The app nails fraction calculations. It got 7/20 overall and got 7/7 on the ones I would have expected it to get. Here’s one it got right.

Math 9 – Equations 

The app got 6/20 overall and 6/6 on the ones I would have expected it to get. It solves basic equations (no logs, no powers, no quadratics, few brackets) correctly every time I try it. Some of the steps seem convoluted to me.

 

 

 

I’m not sure this is the game changer some people fear it is. It’s a calculator, and not a particularly accurate one. As long as we’re asking the right questions, let them use this app. Just have them check their answers on a calculator.

Desmos Classroom Activities and 10 Minutes With Max Ray

Max Ray and I communicated 3 years ago after I was in Philadelphia for a conference. We never actually met while I was there. Since then, he has published a book on problem solving. I’ll confess I bought it, but haven’t read it yet.

We were both at Twitter Math Camp last week. Early on the first morning, I nervously introduced myself to him and said hello. I didn’t want to take up too much of his time, since he was getting ready for his morning session. I intended to go to his problem solving session later on in the camp, but chose differently in that time slot only because there was another one I though might be slightly more relevant to my work next year. Because of that, I didn’t get to interact with him again, which I was regretting.

Then, Max ended up sitting beside me during Eli Luberoff’s keynote and Desmos demonstration. Desmos is more than just an awesome and free online graphing calculator. Eli was having us work through the Desmos Function Carnival activity, which is part of their set of classroom activities that are “Hand-crafted classroom activities. Designed by teachers. Built with love by Desmos.”

I had my laptop open, so Max slid over and we worked together. I’m the type of kid who needs to get everything right, so I was trying my hardest to get stuff done well.

Max said, “Get the next one wrong.” After I got over my initial shock, I remembered where I was and what we were doing. We were evaluating an online classroom activity. In order for an online classroom activity to be useful to students, it has to be built to provide them with useful feedback when they are wrong. I saw what Max was doing and why he wanted to be wrong. We got a few wrong, and we got feedback that would be useful to students. Thanks, Max, for that brief interaction, and for reminding me what I should have been doing in that moment. It made me think I should have gone to his problem solving session.

For the record, the Desmos function carnival appears to be a remarkably well-built activity. Check out David Cox’s video showing what the teacher can see over time as a class works on it. I intend to spend some time this summer working through some others.

 

Golf and TMC

Last week I attended Twitter Math Camp (yes, that’s a real thing). In my conversations with other attendees, we frequently ended up talking about how intimidating it felt being in the company of the other TMCers. This was a passionate and committed group of professionals sharing their best stuff. It was easy to be in awe of some of the presentations. This feeling of awe caused some people to experience angst.

This blog post from Mr. Kent really got people talking. If you haven’t read it yet, head on over there and check it out. I’ll wait. It’s way shorter than this post. Among other things, he says,

To be surrounded by this many people that are this far above me in every area of teaching, learning, growing, intellect, honesty, humor, and kindness, hit me like a stake through my heart. I truly feel as I am nothing compared to those I met here.

I respect how he was feeling. I had moments where I had similar thoughts myself. I’m writing this post to try to make him feel better, at the risk of offending other attendees. I suspect he’s a good teacher.

In addressing this post on her own blog, Kate Nowak says,

I’d just like to say, everybody chill the &^%$ out. We are all good at some things and suck at other things. One thing we all share is the recognition that we all have work to do, and that we can all get better, and that focusing on that is worth our time.

In the comments on Mr. Kent’s blog, Jen weighs in strongly with,

The truth is, many of the teachers at TMC14 have also admitted feeling inferior (look through this morning’s #tmc14 thread). It’s hard not to when so many great ideas are being shared – but remember, these people are there sharing a few great ideas, they can’t all be that awesome all of the time. What makes it harder still is the celebrity reception some of the veterans get from those newer to the mtbos. That’s not reality, and I wish it would stop.

I want to take her last comment a step farther. We need to get over ourselves, as a group. Underlying feelings like Mr. Kent’s, at least when I have them, is the assumption that everyone ELSE at TMC is a superstar. I suspect most of the attendees felt that way at one point or another. Is it possible that all TMC attendees are superstars? We (the attendees) sure act like it is.

I readily accept that the teachers who attend TMC are:

• committed – They do it on their time and 69% pay their own way.
• passionate – They talk about math teaching almost all the time inside and outside the conference. Informal sessions went on in the hotel after the conference was supposed to be over each day
• learners – These are people who want to get better.

Indulge me as I present a golf analogy, if you will.

The local sports radio station is organizing a golf trip to Mexico. The people who go on trips like that are committed (they do it on their time and they pay their own way), passionate (they love that silly game enough to go to all the way to Mexico, which is a long way from Edmonton) and I assume they want to improve their game. I have never been on a trip like that, but I’m fairly confident that some of those committed and passionate players stink at golf.

I’m 22 years into this education game. I’ve worked with hundreds of teachers. In the past five years consulting, I’ve been in dozens of classrooms and observed the teachers working in them. I am fairly confident that most of our teachers are good. I have a theory based purely on observation and my gut that says there are a small percentage of superstar teachers and a small percentage of teachers who really struggle. I sometimes put numbers on those small percentages that range from 2 to 10%, depending on who I’m talking to. I do picture a bell curve with a smallish standard deviation. I have absolutely no scientific data to back up my hypothesis. Personally, there are days when I feel like I’m a little bit to the right of the mean and days when I feel like I’m a little bit to the left of the mean. I rarely feel like I’m off in the far ends on either side. I’m good with that.

I don’t know why TMC attendees would be any different in pure teaching ability than any other set of 150 educators.

At TMC, committed and passionate educators share the best of what they do. Of course it looks good. They don’t share the lessons that tanked (Well, I did, but maybe I’m the only one that ever happens to. See, that self-doubt keeps popping up.). They don’t share the practices they tried that failed. They share their successes.

Let’s go back to that golf trip for a minute. Even the terrible golfers who go to Mexico probably have some elements of the game that they are good at. Maybe one of them knows a good tip for improving putting. Another knows how to correct a slice. Each of them could probably find a thing or two to share that benefits other players. Those things alone, though, aren’t enough for any of our terrible golfers to join the pro tour and make millions. Only a small percentage of golfers are that good.

There are superstar teachers at TMC, certainly. But not exclusively. There are a whole lot of good teachers sharing the best of what they do. Even the best ones there would tell you honestly that they have things to learn. They don’t go to TMC to be in the spotlight. They go to learn, and that’s what it’s about. It’s not about comparing our skills. It’s about growing together.

Professional Development By Teachers, For Teachers

The title of this post is the tagline for Twitter Math Camp (Yes, that’s a thing). What people love about this conference is that the presenters are typically classroom teachers, sharing their best practices. There is real power in hearing about something from a colleague who actually does it successfully.

During the conference, I had numerous conversations with people who talked about their disdain for those outside experts who come in with their theory and crazy ideas and tell us how to teach. There was always an uncomfortable pause when I would divulge that in my regular role, I’m one of those guys. As a consultant, I’m often that outside person coming in to help teachers grow. Fortunately, I’m rarely offended by exchanges like that, because I often said the same thing while I was teaching.

I’d like to make a case for what I do, because there’s power in consulting done effectively. I’ll reframe what I do to give some unsolicited formative feedback to TMC presenters, if I may.

As a consultant, I have time that classroom teachers don’t have to research. I read a lot about education (Books, journals, blogs) so I learn about the theory (and curriculum and standards). When I do workshops, I do my best to connect the theory (and curriculum and standards) to practice . It’s not always possible, and sometimes I deliberately leave it to teachers to make their own connections to their own practice in their own unique situations. Sometimes that drives them nuts. When I quote experts in my sessions, I tend to start with the bloggers who are actually using the strategies I share. Then I go sarcastically to the “real experts”, which are the people who are not teaching, but write books. Both kinds of experts are valuable.

I do demo lessons and lesson studies with teachers. I’m in classrooms a great deal. Sometimes I try things that fail. I’m open about that. Sometimes I try things that work. I’m open about that, too. I rarely  ask teachers to try things I haven’t tried with success myself. I never ask teachers to try things that I’m skeptical about working.

I went to some really good sessions put on by classroom teachers at TMC. Andy Pethan showed a stats activity that totally engaged me. We had to draft an Ultimate Frisbee team based on a set of statistics that we could analyze how we saw fit. Then he used a simulator that he built to have our teams compete. I loved it. I want another crack at that simulator. Defence has to win championships, even in a sport I know nothing about.

Update: In the comments below, Andy provides this link to the code for his simulator. Now I can get my second crack at it!  https://sites.google.com/a/byron.k12.mn.us/stats/projects/ultimate-frisbee-draft/simulator.

Keep your eye on Andy. This guy needed an engaging activity and wrote one, including coding his own Ultimate Frisbee simulator. That’s a cool skill set. He’s going to do neat stuff.

So finally, here’s my formative feedback for TMC presenters. The practices you shared were fantastic. Very few of you connected those practices to theory (and to curriculum and standards). What you did would have been even better if you took just a few minutes to tie it all together. In your presentations, get all consultanty. Just keep that part short.

What impressed me most was the age of some of the presenters. These are young teachers who are not afraid to share their craft with others. That bodes well for the future of education.

Featured Comments

Andy

I wish it were easier to have more flexible roles for education leaders — teach for part of the year, consult / develop software / write curriculum for the other part — so more of us became well rounded with theory, development, and daily teaching practice.

Dan

I’d like more theory also, but I don’t need someone to stand up and tell us that their lesson is an example of embodied cognition or to cite von Glasersfeld or whatever. What I need to know is where they’re coming from. What they look for in a good lesson. What makes a good lesson good for them. What would make their good lesson bad. That’s the kind of theory I need – a personal, theoretical framework.

Meeting the Community

I am just home from Twitter Math Camp (yes, that’s a thing). I’d like to share an overview, aimed more at people from Canada who don’t know what this is than for the people who were actually there.

Twitter Math Camp bills itself as “Professional Development By Teachers, For Teachers.” Last week, 150 teachers, mostly from the US attended a 4 day conference in Jenks, Oklahoma. The venue was the Math/Science building of the Jenks High School campus (yes, you read that right. It’s a campus, complete with a planetarium!). By my count, I was 1 of  6 Canadians and I met one fellow from Britain. I think the rest were Americans. The vast majority of the attendees communicate with each other regularly through Twitter and blogs. Demand for this conference was high. The organizers limited it to 150 and there was a waiting list.

What makes that demand extraordinary to me is that this conference occurs during the summer holidays and most (69%) of the attendees pay their own way. There is no registration fee to attend the conference. Attendees pay travel, hotel and meal costs. The mean age of attendees is 37 and the median is 35.

It was organized like conferences tend to be organized (Keynotes, breakouts, flex sessions), but it felt very different from any conference I have attended before. Keeping it small made it feel more intimate. That the attendees had preexisting relationships made it feel less like a conference, and more like a district PD day. A unique component of this conference is daily “My Favorites” sessions with the whole group together. Teachers get up and share something they do. They are short (5-10 minutes) and diverse in their nature. We saw technology, pedagogy, math, motivation, classroom management strategies and many more.

One of the strengths of TMC is that most of the sessions are conducted by classroom teachers. They have credibility because they are doing the work they are talking about. There were some coaches and consultants presenting (I was one of them). The keynotes were big names (Steve Leinwand, Dan Meyer and Eli Luberoff). As far as I know, they all appeared free, and covered their own expenses. Since no one paid a penny in registration fees, I don’t know where money to pay these speakers would have come from.

Certain things about this conference intrigued me.

• I didn’t see one person walk out of a session. That’s unheard of at conferences. Is it because these sessions were so much better? Is it because people knew each other before and were therefore better able to select sessions that would appeal to them? Is it because of the community feeling and not wanting to offend friends?
• People spent the 4 days together. At a big conference, you may hang out with one or two people, but you don’t connect on this level. At TMC, people learned together all day, and then hung out all evening. Some people didn’t quit doing math. There were spontaneous post-conference math sessions at the hotel every evening.
• I was surprised that at 44, I fit right in. I thought this would be a conference full of 25 year-olds. Regardless of age, everyone was super nice to each other. No one sat alone long at breakfast, or walked too far without someone asking if he could join.
• The organizers must have spent an incredible amount of time putting it all together. It ran like clockwork. There were shuttles, tech support, and social events. They organized hotels and transportation. They did all this on their own time, much of it happening in advance of the conference, while they were teaching.
• It was hard to meet everyone. I did my best, and I met a lot of them, but I missed many people, including several I really wanted to meet. I think I had conversations of varying lengths with over 80 of the 150. I desperately wanted to meet the Sam Shah and Kate Nowak, both of whom were instrumental in getting me tweeting and blogging. I met Sam the first night, and Kate showed up in the same morning session I chose, so I got those two out of the way early. After that, I really enjoyed hanging out the hotel and conference venue and chatting with people.
• I’m not sure I’m as funny in America as in Canada. Some of my best stuff didn’t get laughs during my session. I was pretty nervous. I did a session on formative assessment strategies. Being nervous while presenting is unusual for me. Part of those nerves stemmed from feeling like most of what I was sharing was learned from the people who were sitting in the room. Part of it stemmed from feeling like an outsider. I Tweet a ton, but this conference is billed as PD for Teachers, By Teachers, and I’m currently not a teacher. I didn’t want to sound like one of those annoying experts. More people showed up than I expected, and that added to the nerves. I followed Steve Leinwand’s fiery keynote. The feedback I got after my session was complementary and enthusiastic, which is what we do at TMC. We support each other. I want another shot. I can do better. Next year.

I’m so glad I went. I now have faces and conversations to go with the names of many of the people I communicate with regularly. I had so many conversations I really enjoyed, and not all were about math teaching. I feel like I’m a bigger part of that community now.

 

 

 

My AAC Work

My last post reflecting on my two and a half years at Alberta Assessment Consortium was too much about feelings and not enough about number crunching. Here are some numbers reflective of what I did.

Travel

• Total distance driven = 39 126 km
• Total distance flown = 13 962 km
• Nights in hotels = 97 (48 this school year alone)

Far too many of the drives this winter looked like this:

With all that driving, my 160 GB iPod and its 7824 songs was my best friend.

• Most of the time I play it on shuffle mode, all songs in the queue.
• The top 25 most played is a diverse list including Adele, Biz Markie, Edwin Sharpe, Pitbull, Gwen Stefani, Leonard Cohen, Project Jan & Project Jenny, Shaggy, Mumford & Sons, Band of Horses.
• The most played song (81 plays) was Whale of a Tale by Danny Michel.
• On shuffle, a lot of songs end up coming up that I’m not in the mood for.
• The most skipped song (50 skips) was something called April Showers by Sugarland. I wonder why it’s on my iPod.
• High on both lists are Hate Me by Blue October (39 plays, 37 skips) and A- Punk by Vampire Weekend (37 plays and 37 skips)
• I worked my way through all the Freakonomics podcasts from start to finish.

Cities and Towns Visited For Work

• 29 Different Cities
• Most Visited City – Grande Prairie – 30 Days
• Second Most Visited City – Fort McMurray – 19 Days
• Closest City Visited – St. Albert (or is Sherwood Park closer?)
• Farthest City Visited – Toronto, Ontario

Work

School Visits

• 153 School Visits
• 42 Unique Schools
• Grande Prairie Composite was stuck with me the most, at 23 visits.

Coaching Visits

• 85 Coaching Visits
• 41 Different Teachers Coached

Workshops/Presentations

• Total, including full day, half day, and shorter – 93
• Teachers in workshops – 2017
• Unique teachers in workshops – A subset of that 2017
• Workshops in French – 5
• Most common workshop theme – Formative Assessment, of course.

Meetings

• Meetings Attended – 121 (Ug!)

Demo Lessons

• Total – 43
• Total Flops – 2

Meals With Keynote Speakers

• Steve Leinwand – 1 (But it was actually the second time I dined with him)
• Cathy Lassiter – 1
• Ruth Sutton – 2
• Ken O’Conner – 1
• David Coffey – 1
• Kathryn Coffey – 1

 

 

What I Learned At AAC

My secondment at Alberta Assessment Consortium ends next week. For the past 2.5 years, I have traveled the province conducting a research study in which I worked with math teachers on embedded formative assessment. We also studied the coaching model as a professional learning tool.

As I transition back to my district, I’m reflecting on my time at AAC. I’d like to share with you what I think I took most from this experience.

I could tell you about all the people I met across the province who are doing great things in high school math classrooms, but that would sound trite.

I could tell you about how much I learned about assessment, but I’d have been doing an absolutely terrible job of this work if I didn’t learn a whole lot.

Instead, I want to talk about videos.

When I took the job, I had no idea I would need to make videos as part of the project. The ones I made are posted here. They’re not in the order I made them, but an astute viewer will see my progression. After the first one, we bought new camera equipment because the flip camera wasn’t cutting it. At one point, we had a videographer come in and teach us about cuts, B-roll, transitions, multiple cameras and other tricks. We hired a video “intern”, who made one video for me, and helped me dabble in Adobe. For the most part, though, those videos are all me, and are all iMovie.

The thing is, I had no idea I’d enjoy that creative process so much. Let me tell you how much I enjoyed it.

Last week, I spent a day at a local elementary school filming K-3 students talking about their writing. I hit it with three cameras, one on a boom giving an overhead shot of the students’ work. I recorded an audio track on a separate microphone. I brought a colleague to interview the students so I could focus on filming. I did my best to film it like a pro. In the end, I had more than 90 minutes of footage, filmed from three different angles. This footage is to be used by our video intern under the guidance of future AAC employees to make 30 second snippets to use in workshops and to post on our website.

The thing is, I couldn’t let it go.

Even though I don’t own the footage, and can’t use it myself, I had to make something from it. Knowing full well that no one would ever see it outside our office, I spent hours piecing it all together into something I loved. It’s 15 minutes of young kids talking about feedback. I built in multiple angles. I worked in their funny comments. I worked in their insightful comments. I pieced it all together in a manner that really amuses me. I added transitions and pulled audio tracks from my best track into the clips from the other cameras. I learned how to line that audio up to the students’ lips. It comes in at 15 minutes long, and it’s some of my best work. I’ve revised it twice more after rendering it and showing it to people.

On Friday, I’ll wipe my work laptop clean and pass all my video (including this one) on to the boss on a hard drive. At that point, I won’t even have a copy of this creation any more.

Why did I do all that knowing that very few people would ever see it, and that I couldn’t keep it? Because it reflects the thing I learned most about and really enjoyed doing during this job. Who (other than that Bloom guy) knew that a creative process could be so enjoyable and valuable? That’s a nice thing for a rigid math guy to come to understand.