Guest post: Read the methods first
Originally a comment by Claire on Researchers found.
I can’t access the paper because it’s behind a paywall and I’m not at work. The appendix has the methods and detailed description of how they collected the data and that’s all I really care about. I always read the methods first. If I think the methods are garbage then the paper is garbage and I can devote my valuable time to something else.
PNAS is a good journal and I’m a little shocked that the paper was published. Statistically, this paper is flawed in many ways. Firstly, none of the methods adjusted for confounders. Confounders are elements that you have not accounted for in a study that may be coincidentally correlated with the trait of interest.
Here, they report that an overwhelming majority of parents identified as “liberal”. This is a problem in a study like this. I’m sure you can all see it already, parental attitudes to the trans movement are highly correlated with their overall political stances. Conservative parents are much more likely to disapprove of any expression of difference in gender presentation and sexual orientation, even at a young age. So you have already introduced selection bias right from the beginning. It’s right there in the data, you can see it.
Next confounder: locations of recruitment. This is a problem in all studies, including the sort of work I do. But I adjust for it! Cities are more liberal than rural areas as well as being more populous (easier to recruit in big cities vs small communities) and San Francisco is very different to Oklahoma City, politically speaking.
There are others, but I think I’ve made my point.
The methods themselves are terrible for these kinds of problems. Tests like t-tests, chi-sqs and even the more complex tests like ANOVA are not capable of adjusting for confounders (ANCOVA would work, but they didn’t use it).
There are other problems with the statistics too; some of the tests are inappropriate because of the “small cell” problem, they can’t report odds ratios or betas because they didn’t do the right tests. But I don’t want to get into the weeds here.
Finally, they don’t seem to have the faintest idea of how hypothesis testing works. They state their null hypothesis (H0) and their alternate hypothesis (HA) as two separate hypotheses (1a and 1b and so on). This indicates a lack of understanding of what they are doing. They list several hypothese in this paper (not counting the whole weird H0/HA presentation) and this hurts them.
Statistical power is the probability you will correctly reject the null hypothesis, i.e. the alternate hypothesis really is true. But every time you add another test, you have to adjust for it, which reduces the power. Here they claim good power but do not present their power calculations.
I’ve only outlined a few of the most pressing issues with it; there are more but I don’t want to bore you all. This is almost certainly a terrible paper, based on the methods description. I tried to be as open and unbiased as I could, reading it as if I were a reviewer of a paper in my own field. If I had reviewed this paper, I would have been very concerned about the standard of statistical expertise in this study and probably written to the editor to ask it be improved before I was even willing to do the review.
Am I just stupid? What if this study had an impeccable methodology, accounted for all potentially confounding variables, and arrived at conclusions of statistical significance? What is that the study wound have found?
That some kids prefer the toys, clothing, etc. typically associated with the opposite sex?
It purports to say something more than this, though, right? I mean, it must.
The study doesn’t show anything we don’t already know — it’s not even remotely news that some kids show “strong preferences for toys and clothing typically associated with” the opposite sex. (Or, as they dishonestly put it, with “their gender identity, rather than their biological sex.” I’ll get back to that sneaky little bait-and-switch in a moment.) This study takes that ordinary piece of No Shit, Sherlock and frames it around the just-invented-yesterday concept of “gender identity,” separating the kids whose liberal, city-dwelling parents declared them to be “trans” from the kids who didn’t get that treatment, and found — lo and behold! — the kids the parents transed have a tendency to be gender nonconforming (yeah, no shit, that’s why the parents transed them in the first place)… and (this is the important part) that this is proof that their “gender identity” — i.e., their status as “truly” trans — is innate and unchangeable. Which is absolutely batshit.
They are trying to argue that because gender non-conforming behaviour can sometimes be identified relatively early — that gender-atypicality is to some degree inborn — that therefore “trans-ness” can be identified early — i.e., that they’ve found evidence that “transness” is inborn. It’s completely circular: they just took a bunch of gender-atypical kids whose parents already decided that their gender-atypicality was a sign that they were trans, and said, “See? We have proof these kids are trans: look how gender-atypical they are!”
To put it another way, they’re pulling a bait-and-switch: they’re smuggling the concept of “gender identity” into science by sneakily pretending it’s synonymous with gender-nonconformity.
Well, that depends on what you mean by “gender.” If you mean, some kids show a strong preference for gender-atypical activities despite social influences pushing them to gender-conform, then, sure. But that’s not remotely news. This study didn’t just now discover the existence of tomboys. But what they’re doing here is taking the word “gender” and slooooooowwwly morphing its meaning from something like, “how close a child adheres to sex-based social norms” (as in the quote above) to “what the child’s true identity is” …
…and then, eventually all the way to defining the word “gender” as synonymous with whether a child truly is or is not innately “trans”:
Less “supportive” environments?! “Poisoning the well,” as Ophelia would say.
You could easily rewrite that conclusion in a way that has absolutely nothing to do with “gender identity” (here camouflaged as simply “gender”) or “trans.” Here:
Oops, that last sentence was supposed to be outside of the blockquote. I miss the “preview comment” button.
The other thing is, these are parents who are liberal, and affirming of the trans identity, which means they may not have pushed the gender conforming toys in the first place. When I was a kid, it was sort of a mocked stereotype by people on the right, in the conservative Christian community, (read: my parents) that liberal parents were screwing up their kids by not teaching them which toys were ‘proper’ for their sex (my parents wouldn’t have dreamed of using gender in such a nonsensical way).
So, the kids are given freedom to play with whatever toys they want. ‘Boys’ toys are often more fun than ‘girls’ toys. The kids select the toys they like the best; they do not fit the stereotypical expected toys of the observed sex; the parents drank the trans KoolAid while in college, and are fully woke; they shriek, ‘look, she’s a boy!’; the kids are given new names and new pronouns, clothed as the sex their parents believe they are; a researcher comes around wanting to support trans rights; voila. Instant gratification for all…except the poor kid who will perhaps find themselves unhappy later in life because all they really wanted was to play with the fun ball (and whoever said a ball has to be a boys toy, anyway?) rather than the stupid doll, or because they wanted to play with the pretty Barbie instead of the boring car.
It’s very circular.
I don’t know how the study authors know they are studying “trans kids” at all. At 3 years old! They don’t know what they are talking about.
Use of “assigned sex” screams “AXE TO GRIND.”
@Ben
No, you’re not stupid, you’re just asking a different question. I was only concerned with the methods they used to analyze the data, which prima facie should have precluded it from publication.
I should upfront admit that this is outside my area of expertise. I am not a social scientist. And this is going to be another long post, sorry.
To your question, I think the answer is no. The study design is terrible, the questions they are asking are poorly defined and the data collection methods are so awful, I don’t think this study can be rescued. There’s a limit to what statistics can do. This really is a situation of bad methods and bad data. I could write a novel on just how many bad things are in this study. But let me give you a snippet.
This was an outcome measure. You don’t have to be a social scientist to see the problems with this. As a little girl, I lived almost exclusively in jeans and pants with shirts in my favorite colors, blue and purple. How would I be scored on the above scale? Wearing jeans isn’t a gender marker, everyone wears them and they’re convenient for parents because denim is sturdy. This means there is a skew in the data, boys do not have the same option to wear skirts or pink shirts if they choose without inherently being scored as a 1 or a 2. This one is hard to adjust for, almost impossible.
But let’s put that aside and instead imagine the data collection and statistical methods were perfect. First of all, you cannot perfectly adjust for all confounders, because a) you can’t always measure them perfectly and even if you could there is a limit to how much you adjust before you swamp your signal and b) there are some you don’t know about. That second problem is bad because you can’t measure or adjust for what you don’t know about. And there are always confounders you don’t know about. Literally always.
But even without those issues, let’s say that the confounders are not significant and the data problems are resolvable (they’re not, btw). What is the question the authors are actually asking. It’s not always the one researchers think they’re asking. I’m going to pick out one.
Ignoring the horrific way this is written, they think they’re asking if trans kids will remain trans as adults. But they’re not. They’re actually asking if the kids think they will be trans as adults. The answer to which is, so what? These kids are 3-12 years old. You might as well ask them if they think they will be an astronaut when they’re grown up.
tl;dr No. This study will never mean anything, no matter what they do to fix the technical aspects of it because the data are inherently flawed.
Plus, as I’ve mentioned before, the question of whether extremely gender-nonconforming kids will grow up to be trans is well-covered terrain and the science is settled: they almost certainly won’t, unless they’re socially transitioned at an early age, in which case they almost certainly will. It’s like when the tobacco lobby kept commissioning more and more studies into whether tobacco causes cancer. We already know that they do. It’s just a dishonest lobby pushing a dangerous agenda.
Of course, what they are really asking is whether children who do not fit stereotypes have been identified as trans. They just don’t realize that.
Part of the problem is identification of gender-specificity. Clothes? Huge problem, especially in girls, who wear clothes that are now being coded male simply because they do not fit an archaic version of ‘girl’s’ clothes.
Toys? Even worse. Ball – boys. But both girls and boys play with balls, and there is no inherent reason for it to be boys. Cars – boys, because even at a young age, we are told, children associate cars with their fathers. Why? They probably see their mother drive as much as their father – more, if they are like most families where the mother hauls the kids way more than the father. A lot of fathers don’t work on cars or mess with cars, and a not insubstantial number of mothers do. So there is a confounder, too. Why in the world would a child associate cars with the parent they are less likely to see in the car, at an age before they are old enough to realize Daddy is the one working on the car, or in a family where Mommy is the one that works on the car? Or neither parent? That answer is simple – it isn’t inherent, because cars haven’t been around long enough for that to be in our DNA. It’s movies…TVs…video games…books…and especially advertisements. Cars have been socialized as a guy thing.
And then there was the thing posted here in the past, where a doctor was telling parents if their daughter refused to wear barrettes and dresses, she was probably pre-verbal trans. Why not a pre-verbal feminist? Why not a little girl that just wants to be comfortable in her clothes, be able to play without worry, and like the feel of her hair blowing free?
All hail the patriarchy. It is now wearing the colors of progressivism.