I suspect that some portion of the blank slate people are just statistically weird.
introduction
A lot of blank slate arguments go something like the following "We only like [topic at hand] because that's what we're taught" For example, girls only like playing with dolls because young girls are given dolls to play with. Or, men only like sexualized images of women because that's what we teach as a society. etc. These kinds of arguments can sometimes be true, but I also see a lot of cases where they're either over-extending or downright baffling.
the basic premise
My own experiences have led me to suspect that at least in some cases, the proponents of these arguments just fall out of the median range for whatever trait is being discussed. As a great example, we had a friend who was something of a polyamory activist. Her belief was that everyone would be polyamorous if we could just learn to give up our jealousy. In her view, most normal people were just defective polyamorists. I heard a darkly comedic example of this frame of thought when listening to a podcast once. Some poor young man had accepted the argument on its face, and was attempting to "get over his ego and his jealousy" by letting his girlfriend go on a date with another man. The man had turned to a subreddit described his experience: he was having such an extreme panic attack that he was on all fours in his bathroom, hyperventilating and dry-heaving. A well-intending idealist had led this poor man astray.
what I think is happening
So, I referenced my old "Enthusiastic Dissonance" argument earlier, and my own weirdness here had also previously led me to a lot of bizarre conclusions. eg: "people can't really enjoy going to crowded concerts or crowded restaurants. They must just be pretending to enjoy it because everyone else has already been taught to enjoy it. Obviously no one really likes this, but we're sort of stuck in a collective action problem." These sorts of conclusions were degraded by real-world experience, of course, and the simpler thesis became clear: I just don't like crowds or loud noises, but most people do.
So what do I think is going on? I think my polyamorist friend was just different. She probably really does not get jealous. And I'll bet she saw all the normal sort of dating rivalry and jealousy around her, and just accepted it. And then as she began dating, those feelings never landed, they never made sense, she never experienced them. To her, I think it felt like society had tried to teach her to be jealous, but she was able to think her way out of it. Now, I think this is incorrect; she didn't possess sexual jealousy in the first place, so the experiences of other people just never made sense to her. I can see how it felt like she was being "taught" to be jealous by society. But she wasn't being taught anything; most people just experienced jealousy, and she didn't. Just like me with crowded concerts: I felt like I had to be taught to enjoy them, because intrinsically they were really unpleasant experiences for me. Since everyone else enjoyed them, I naively assumed they just must have learned the lesson more easily.
We learn so much through social accretion, that it's easy to see how anyone could make this mistake. Even the most independent-minded among us still feel a pressure (both internal and external) to adopt the beliefs and attitudes of those around us. When the vast bell curve of people differs on some trait, we will feel an implicit (and again, sometimes explicit) pressure to adopt the trait ourselves. When it's really anathema to us, it just feels like society pressuring us. And most of us, even the really talented extroverts, use our native perspective when trying to model other people. This deserves an essay of its own, but it is a fundamentally surprising and persistent gap. It explains a lot of things, such as why men and women misunderstand each other so often. I think it also explains our case; we differ in some key trait from most people, so when we try to understand how people experience that trait, we are biased by our own perspective, and try to explain it in those terms. eg: "People must have been taught to like crowded restaurants, because they're clearly super unpleasant!"
consequences
I think this problem has interesting consequences for social values and public policy. We're often optimizing for the minority. And I don't mean a racial minority or anything. I just mean a statistical minority. We're optimizing, at least in some cases, for a loud, vocal subset of people whose best interests are actually opposed to the wider bell curve of humanity. Imagine if my polyamorist friend, a well-intending idealist, got to set marriage law or set social values. More people overall would be less happy and less satisfied. And it would sort of be an accident. She would have just been trying to help, and "liberate" us from our base jealousy.
Or, imagine a world designed for me. Hard limits on how crowded or noisy a restaurant is allowed to be. Night clubs totally banned. Hard noise and population limits on outdoor concerts. I might personally be happier, but the world overall would be less happy. I also want to highlight that there are cases where it's good to optimize for the minority: disabled access to buildings, services, etc. Translation services for government functions. There are more cases than this, but I do want to acknowledge that this is not always a bad move. I just think we need to be clear-eyed about when and why we're doing so.