Patheos

BuddhistCatholicContemplativeEvangelicalGeneral ChristianHinduJewishLatter-day SaintMuslimNew VisionsProgressive Christian

Beliefs

Topics

Columnists

Resource Library

Research Tools

Video

Podcasts

Thought Readers

Faith In Media Tracker

Newsletters

Ad-Free & Subscriber Log-In Subscription Support

Patheos

Ad-Free & Subscriber Log-In Subscription Support

Follow Patheos Patheos on Facebook Patheos on YouTube

Love, Joy, Feminism

Guest Post: Methodological Problems with Brian Ray’s Study on Youth and Religion

Last updated on: September 9, 2013 at 4:08 pm   by Libby Anne

Guest Post: Methodological Problems with Brian Ray’s Study on Youth and Religion 2013-09-09T16:08:53-04:00 Libby Anne

LOGO Patheos Explore the world's faith through different perspectives on religion and spirituality! Patheos has the views of the prevalent religions and spiritualities of the world.

A Guest Post by Apodosis 

Libby Anne posted recently about a new survey conducted by Brian Ray of the National Home Education Research Institute, a non-profit which conducts studies of questionable scientific validity on homeschooling. As a Ph.D. social scientist myself, I looked over the new survey with a critical eye and I’m sorry to say there won’t be much useable data gleaned from it because it is rife with methodological problems.

The survey has over 100 questions and nearly every question needs revisions, so I’m just going to summarize my main critiques of the study.

1. Institutional Review Board

When you are conducting a survey that collects personal information from the participants, every IRB in the country requires that you have signed informed consent forms for each of your participants. In online surveys, these tend to include a cover page describing the purpose of the study and what is asked of the participants, as well as a signature box to show you have understood your rights.These forms often include the phrases “You may stop the survey at any time without giving a reason” and “You are not obligated to answer any question you don’t want to”. They provide contact information for the PI and other associated researchers, as well as for the IRB that approved it. “If you feel your rights have been violated, call this number” etc.

In the Gen 2 survey, there is no such information. It is not clear who is being asked to participate (see #2), the goals of the study are not honestly stated (see #3), and there is no contact information for the PI or an IRB to contact if you feel your rights have been violated. I had to dig on NHERI’s website to find the email address of the PI, Dr. Brian D. Ray (it’s [email protected], btw, in case you feel your rights have been violated), and he actively discourages you from contacting him. Red flag.

A peer-reviewed publication will not publish any results from research that was not overseen by an IRB.

2. Target Population

It is not clear who is being asked to participate in the survey. In the “About” section, Ray says the participants are “those between the ages of 18-38 years old that grew up in religious homes”, but in the FAQ he says “Anyone between the ages of 18-38” may participate. I am between the ages of 18 and 38; I was raised in a moderate mainline Protestant family, and I am now a progressive mainline Protestant. I honestly cannot tell if he wants my data or not. Though both of those statements about who should participate apply to me, the questions on the survey indicate otherwise.

3. Goals

Which brings me to my most serious methodological critique of the study, which is that the goals are not honestly stated. There are two different studies conflated here which have entirely different goals. Dr. Ray even alludes to this in the “About” section: one goal is “to come up with data points of key influences that either encouraged or deterred the participants from practicing the same faith as their parents”; the other goal is to “use the statistics from this survey to help equip parents to make more informed decisions in the education and spiritual guidance of their children.” That is, in simpler terms, the goals are (A) to find out how young people’s religious views change as they reach adulthood, and (B) to figure out how to make sure young fundamentalists/evangelicals stay in the fold.

In fact, the main goal of the study seems to be (B) with a shallow veneer of (A) superimposed on it. Now, (A) is an interesting study whose results I would look forward to reading. (B) is not a scientific study. You do not perform a scientific study with the goal of achieving a certain result. Social science is about trying to describe and explain human behavior, not about trying to change it or attach value judgments.

Here are some examples of the problems that arise when the two studies are conflated.Dr. Ray pays lip service to the idea that young people may belong to a variety of faiths, as evidenced by his questions:

Generally, what kind of religious service did you attend as a child?

What kind of church or religion do you currently associate yourself with?

Available answers include Atheist, Agnostic, Buddhist, Hindu, Jewish, Muslim, and a variety of mainline Protestant denominations in addition to the exhaustive list of evangelical/fundamentalist groups. However, on the next pages he asks questions like

My current church is serious about applying the biblical principles of eldership, shepherding, mentoring, and church discipline.

Did you ever get any “worldview” training?

How often did your Father/Mother read the Bible to you?

These questions indicate that the primary audience of this study is people raised in a fundamentalist/evangelical/Christian patriarchy home. How are Buddhists and Hindus and Atheists supposed to answer these questions? Note that there is no “Not applicable” option available. This brings me to #4.

4. Not Applicable

It is important in surveys to allow your participants the freedom to answer the questions honestly, and not be forced to pick the “closest” answer. Many of the questions follow the “Have you stopped beating your wife” pattern, where e.g. a person who has never beaten their wife has no honest way to answer the question. “Not applicable” is only an available answer for one of 100 questions in the survey. This becomes a problem with questions like the following:

How distant or close do you feel to God most of the time?

What was the status of your parents’ marriage during your raising?

My father was very involved in our family home life.

How would an atheist answer the first question without “Not applicable”? How would someone answer the second question if they were raised by two parents in a committed relationship who were not married? And in the third case, what if you didn’t have a father?

“Other” followed by a fill-in-the-blank should also occur much more frequently as an available answer in questions such as:

Have you, or your significant other, ever chosen to have an abortion? Yes/ No

Do you think that people should wait to have sex until they are married, or not necessarily? Yes/ Not necessarily

How would you describe yourself when you were a child? I was very rebellious/I struggled with rebellion, but overcame it/I was always fairly obedient and honoring as a child

In the first question, it would be more appropriate to ask “Have you ever had an unwanted pregnancy?” and then to ask how it was dealt with. As it stands, a person who has never been pregnant and a person who has carried a rapist’s child to term would give the same answer. In the second and third questions, there are clearly more than just the possible answers given. My answers would be “No, unless they want to” and “My parents were supportive of what I chose to do and who I chose to be”.Participants in the study need a way to answer these questions honestly, so even if Dr. Ray wants to limit his variables it would be more methodologically sound to provide an “Other” response with a fill-in-the-blank to such questions.

5. Leading/Biased Questions

This brings me to another point: almost every question is leading or has other problems caused, for the most part, by faulty assumptions and a lack of imagination on the part of Dr. Ray.

“What kind of church or religion do you currently associate yourself with?” Some people consider themselves to be a member of a faith without having a specific faith community. Some people consider themselves to be members of multiple faith traditions. A better question would be “How would you describe your religion or faith?” with multiple check-boxes allowed. And he should list all the religions, not just the ones he could think of off the top of his head! He forgot at least (off the top of my head) Baha’i, Sikh, Wicca, Christian Science, Animism, Deist, Shinto, Unitarian Universalist, Not applicable, Other____, and he did not provide any different denominations of Judaism or Islam.In addition, he should have grouped the religions together by type rather than alphabetically—this list makes it very hard to see if your faith is represented. And groups like “Mormons” and “Congregationalists” should be called by their actual names—LDS and UCC, respectively. This list of religions, while it pays lip service to religious diversity, is actually offensive in its exclusions and ignorance. Plus, there is no acknowledgment in this survey of mixed-faith households. What if your parents were of different faiths, or what if you and your significant other do not share a faith?

Questions such as

What is your sex/gender? Male/ Female

How was your relationship with your Mother / Father when you were 16-17 years old?

How often did your Mother / Father explain biblical principles to you?

presume an oversimplified, heteronormative view of gender and family.Sex is not the same as gender; there are quite a few other genders than just male and female; you might have parents of multiple genders or the same gender (and therefore not have a “Mother” and a “Father”); and you might have been raised by other family members, or primarily by friends, or in foster homes.If this was really a survey about changes in young people’s religious views, it would try to get an accurate picture of their lives without limiting them to these binaries. Ray should also ask about people’s sexual orientations if he’s that interested in the status of their romantic lives; however, his reference to “homosexual” “encounters” in one of the questions indicates that he probably does not believe in sexual orientation as a concept (see #6).

6. Limited Mindset

Many of these questions portray a mindset that is isolated to the evangelical/fundamentalist/Christian patriarchy/Quiverfull/Purity movements–it’s possible that Dr. Ray does not even realize there are other ideologies out there. For instance, there is this question/answer set:

What statement most aligns with how many children would you like to have? I don’t want to have children / I want no more than a few children / As many children as God will provide / I don’t know

This really should be divided into multiple questions. First, he should ask “Do you plan your pregnancies?” If no, he should ask about “how many children you hope God will give you”, but if yes, he should ask how many you plan to have. I don’t think Dr. Ray realizes that some people plan their pregnancies. Then there is the following question/answer pair:

If you have—or were to have—children, what form of education do you plan to use for them? Christian school / Christian school and homeschool / Christian school and non-Christian private school / Christian school and public school / Homeschool / Homeschool and non-Christian private school / Homeschool and public school / Private school, non-Christian / Public school / Charter school/virtual charter / Other

At least this one has “Other” as an option, though it doesn’t let you write in your response. First of all, it’s a badly designed question—he should just list the types of school and let you check as many boxes as you want. Second, the question relies on the assumption that parents would choose their children’s manner of schooling before the children even exist. What if the child is gifted or disabled? Would that change the parents’ plans? And Dr. Ray does not even realize that people who would answer like me are out there—“It depends on the child’s needs and wants, as well as other considerations such as expense, distance, quality of education, etc.”

Then there’s this question/answer pair:

Did your parents use corporal discipline (spanking) with you?No / Yes, consistently and they were generally under loving control / Yes, consistently and often they were not under loving control / Yes, inconsistently and they were generally under loving control / Yes, inconsistently, and often they were not under loving control

These answers presuppose a worldview where the value of spanking is not open to debate (spoiler: we don’t live in that world). It would be more accurate to ask “What was your parents’ position on spanking?” “How often were you spanked?” “Are your attitudes toward it positive, neutral, or negative?” “Would you spank your own children?” etc.

And then there’s the part where he puts these two questions next to each other, clearly conflating them (shades of Libby Anne’s two boxes): “Have you had a sexual encounter or physical relationship with someone to whom you are not married?” and “Were you ever sexually abused before age 18?” Note that a girl who was repeatedly raped by her Christian patriarchy father for ten years and a girl who was raped once by an acquaintance at a party as a teenager would probably have different reactions to their faith traditions, though they would give the same “Yes” response. No effort is made to make this distinction in the survey. And nowhere does the survey ask about physical abuse, since Dr. Ray probably doesn’t think it exists.

Or how about the question “If you were unsure of what was right or wrong in a particular situation, how would you decide what to do?” with the available answers: Do what would make you feel happy / Do what would help you to get ahead / Follow the advice of a parent or teacher, or other adult you respect / Do what you think God or the scripture tells you is right / Something else. These answers reveal Dr. Ray’s belief that morality does not exist outside (his brand of) Christianity.

7. Exclusionary Language

The language Dr. Ray uses is exclusionary and often confusing. For instance, he persists in using terms like church, pastor, scripture, prayer, Bible, youth group, Sunday School when these terms are uniquely Christian and do not apply to people of other faiths–he ought to say faith community, religious leader, holy book, prayer/meditation, religious education if he wanted to get accurate data. He also fails to define several terms which I don’t understand because I was not raised in a fundamentalist environment: family-integrated, homeschooling-friendly, shepherding, church discipline, worldview training. In his questions about belief in “God” and “heaven”, he ought to ask separate questions about each property he wants to assign to these words’ meanings (e.g. “Do you believe in a deity or deities?” “If yes, do you believe the deity/deities is/are omniscient? Omnipresent? Omnipotent? Does it/do they have a gender? Does it/do they affect everyday events?” etc.). If Ray wants accurate data, he should define confusing and ambiguous terms.

You should not be able to tell someone’s political and religious beliefs from survey questions designed to elicit yours. You should not be asked to give dishonest answers to survey questions because your honest answers are unavailable as options. You should not have to infer how exclusionary language in the questions would apply to your situation. You should not be asked to give out your personal information without giving your informed consent. An IRB would not approve this study. I would not rely on any of the conclusions that come out of it.

Homeschool

Brain Ray

guest post

Previous Post



July 18, 2013

My Brothers Too: Purity Culture and Double Standards

Next Post



July 19, 2013 CTBHHM: Earning Your Abuser's Trust


Browse Our Archives

Select a Category Atheism Evangelicalism/Fundamentalism Family Feminism Homeschool Meta Parenting Patriarchy Politics Purity Quiverfull Social Justice Uncategorized Worthwhile reads

Select a Month February 2021 January 2021 December 2020 November 2020 October 2020 September 2020 August 2020 July 2020 June 2020 May 2020 April 2020 March 2020 February 2020 January 2020 December 2019 November 2019 October 2019 September 2019 August 2019 July 2019 June 2019 May 2019 April 2019 March 2019 February 2019 January 2019 December 2018 November 2018 October 2018 September 2018 August 2018 July 2018 June 2018 May 2018 April 2018 March 2018 February 2018 January 2018 December 2017 November 2017 October 2017 September 2017 August 2017 July 2017 June 2017 May 2017 April 2017 March 2017 February 2017 January 2017 December 2016 November 2016 October 2016 September 2016 August 2016 July 2016 June 2016 May 2016 April 2016 March 2016 February 2016 January 2016 December 2015 November 2015 October 2015 September 2015 August 2015 July 2015 June 2015 May 2015 April 2015 March 2015 February 2015 January 2015 December 2014 November 2014 October 2014 September 2014 August 2014 July 2014 June 2014 May 2014 April 2014 March 2014 February 2014 January 2014 December 2013 November 2013 October 2013 September 2013 August 2013 July 2013 June 2013 May 2013 April 2013 March 2013 February 2013 January 2013 December 2012 November 2012 October 2012 September 2012 August 2012 July 2012 June 2012 May 2012 April 2012 March 2012 February 2012 January 2012 December 2011 November 2011 October 2011 September 2011 August 2011 July 2011 June 2011


Follow Us!

Follow Us on Facebook Follow Us on Twitter



TAKE THE

Religious Wisdom Quiz

Ecclesiastes 3:1 says, "To everything there is a ______."

Select your answer to see how you score.

Blessing Season Time limit Purpose

Related posts from Love, Joy, Feminism

Nonreligious

Homeschooled Congressman Urged Conservatives to "Lightly Threaten" Their...

Love, Joy, Feminism

Nonreligious

This Homeschool Program Replicates the Office, Not School

Love, Joy, Feminism

Nonreligious

Blogger: COVID-19 Has Shown SAHMs and Homeschooling Are...

Love, Joy, Feminism

Nonreligious

Homeschool Girls and Elizabeth Bennet's Subtle Acts of...

Love, Joy, Feminism

TRENDING AT PATHEOS Nonreligious

Dethroning Jesus and Enthroning Yourself

Jesus Never Asked to Be Worshipped. He Asked to Be Understood. Christianity...

Why Most Christians are Really Paulians and Jobians

I Don't Care, But It Matters Let's get this out of the...

Essential Tips to Successfully Navigate the Selling Process

Selling a house can feel overwhelming, but it doesn’t have to be....

To Make Remote Work Effective, Think About Others

The COVID-19 pandemic forced many companies to rapidly transition to remote work....

Flexibility Is Key to Solving Problems of Tomorrow

The world of work is undergoing a dramatic transformation. The COVID-19 pandemic...

view all Nonreligious blogs

Patheos on Facebook Patheos on YouTube

Ad-Free & Subscriber Log-In Subscription Support

Copyright 2008-2026, Patheos. All rights reserved.

Subscription Support Ad-Free & Subscriber Log-In

Follow Patheos Patheos on Facebook Patheos on YouTube