blighttown

Does it matter if the Marshmallow Experiment did not replicate?

The title probably makes the case a bit too strongly. Per the Institute for Family Studies:

"So, yes, the effect [of the revisited experiment] is considerably smaller—which is unsurprising in a replication, especially when the methods are different—but real and statistically significant. In general, kids with better self-control at age four do better academically at age 15."

In the same article, they point out that The Atlantic makes the the opposing case (sort of):

Affluence—not willpower—seems to be what’s behind some kids’ capacity to delay gratification.

In other words, it's not clear that the Marshmallow Test actually failed to replicate. But per some of the discussion found online, it's not obvious to me that the Marshmallow Test matters very much either.

I participated in a brief discussion which conveyed some dismay that the Marshmallow Study had not replicated. At the time, I was unaware that "failed replication" was probably too strong of a case. I'm sure that The Atlantic's narrative had permeated the culture and informed both of our points of view.

My initial impression aligns somewhat poorly the discourse I was able to dig up. I felt (and still feel) very strongly that impulse control must be a major indicator of success in life. It's almost impossible to me see how this could be debatable. In my own life, my impression is that my impulse control is badly outpaced by my intelligence. It's important to clarify that this really is not a compliment regarding my intelligence; rather I feel unbalanced. I understand what sort of actions I could take on a daily basis to be more successful. But, how much success do I really have with regard to dragging myself to perform conscientiously on a day-to-day basis? Not nearly as much as I'd like.

And so in my eyes, whether or not the Marshmallow Experiment replicated says far more about the methods and assumptions of the experiment than it does anything about impulse control. Impulse control can improve with age. (but perhaps that's taken into account, some kids just start ahead of other even if almost everyone improves linearly?) It's also not clear that to me that more marshmallows actually constitute a significant enough reward. At least for me personally, there's a cost to exercising impulse control and the trivial things often don't meet the threshold. Maybe kids just like being defiant? Maybe terrible impulse control at a young age doesn't always map clearly to impulse control at an older age? (in a large enough group, this problem should be avoidable. ie, if it's true on average then the effect should show) Maybe some kids were anxious about the testing environment, and their anxiety is what eroded their otherwise competent impulse control? I could go on.

Really, the entire test feels like an annoying "just so," story. does one failed instance of impulse control at a young age really determine much of anything? This is a different question than "does poor impulse control at a young age really determine much of anything?"

I hadn't intended to borrow so heavily from Institute for Family Studies, but they make a lot of the points I'd intended to discuss. (and do a better job of it) Notably, they claim that:

The way I’d put it is this: The new data turn the classic result, long treated as a simple and effective demonstration of the power of self-control, into a Rorschach inkblot. Whatever theory you have about why some kids do well academically and others fail, you can read the data in a way that supports it.

This author makes the claim that this is something of a cultural shibboleth, and there seem to be a few questions at stake:

I haven't seen anything in the followup commentary which really speaks to this in a legitimate way. My personal opinion is that The Atlantic article is mostly just performing ideological wish-casting. The Atlantic author makes a number of points which don't actually speak to whether or not impulse control is innate:

for example, Ranita Ray, a sociologist at the University of Nevada, Las Vegas, recently wrote a book describing how many teenagers growing up in poverty work long hours in poorly paid jobs to support themselves and their families. Yet, despite sometimes not being able to afford food, the teens still splurge on payday, buying things like McDonald’s or new clothes or hair dye.

Is this because of stress? Cultural factors? The innate conditions of poverty? Genetics? The article doesn't speculate, and so all we have is a description of behavior, rather than any sort of explanation. The article continues:

at Indiana University, we found that low-income parents are more likely than more-affluent parents to give in to their kids’ requests for sweet treats. These findings point to the idea that poorer parents try to indulge their kids when they can, while more-affluent parents tend to make their kids wait for bigger rewards.

Again, both of these points could be read to simply mean that the low-income parents have poor impulse control. It really does nothing to address whether their circumstances or their genes have saddled them with low impulse control. To be clear, I'm not actually making either case here, but the quality of discourse is fairly disappointing.

To that point, it seems obvious that we need better measures of impulse control. Does it vary over a lifetime? What does lack of sleep or stress do to impulse control? Are there different ways to measure it? Is "impulse control" really a singular concept when you get down to the actual biology of the brain? Measuring it solely based on whether a child eats a marshmallow seems silly. (and annoying) And to the extent that any psychological traits are genetic, they are most certainly polygenic. In other words, we're not going to find the impulse control gene anytime soon, and I doubt we'll have anything which will settle the nature vs. nurture debate here.

It's also not clear to me what the Marshmallow Experiment would have done for parents practically speaking. It cannot possibly be the case that prior to the experiment people did not understand the value of impulse control. And if impulse control is trainable, then you don't need the experiment to make this point, and the experiment does not really lay out how you'd successfully train impulse control. Conversely if impulse control is innate, then parents cannot do anything with the experiment but potentially set their expectations low (or high, depending on the outcome) right from the start. (and this seems like an obviously misguided strategy.) So, does the experiment matter at all? Did it tell us anything we didn't already know? And, does it actually resolve questions such as nature vs. nurture? Can parents do anything with the findings of the experiment? I'm just not convinced the experiment matters at all.