Arguing with the survey
All the major surveys sponsored by the 19 large city Jewish federations since at least the 1960s have been based on probability samples. Until now.
I write as a sociologist and long-time student of American Jewry, deeply troubled by the newly released Jewish Federation of Northern New Jersey survey, because it was based on a sample that is not representative of the area’s Jewish population.
All the study findings represent the attitudes, behaviors and demographics of only the people who completed the questionnaire. This self-selected sample does not represent data about the entire Jewish population in North Jersey.
Get The Jewish Standard Newsletter by email and never miss our top stories Free Sign Up
Among the major segments of the community improperly represented are the unaffiliated, young adults, college students, singles, elderly, poor, intermarried, recent emigrants, cultural Jews and the vulnerable. That is, segments to which the Jewish community often seeks to service in outreach programs. The fact that some respondents to the survey are included in the above-mentioned groups is irrelevant, as they are a biased sample of those groups. The new study does not provide reliable data to help with planning to meet the needs of any groups.
The federation incorrectly contrasted marketing and demographic surveys. While the emphasis of studies can vary, this is a false dichotomy in the case of federation studies. All federation surveys for decades have collected both marketing and demographic data. The primary difference between the new survey and all the other major federation-sponsored surveys is methodological. The Northern New Jersey federation intentionally decided to sponsor a survey with a self-selected sample, knowing that respondents would not be representative of the area’s Jewish population, to save money.
Everyone wants the federation to save money. However, this shortsighted decision is a case of penny wise and pound cheap. The biased data now available are likely to be used by some Jewish organizations to make improper decisions, which will be vastly costlier to those organizations and the wider Jewish community in the long run then the relatively small amount it would have cost to do the survey using a reliable sample.
Federation professionals reported at one of the public presentations that the survey cost the federation $160,000-$200,000, while a survey with a representative sample would have cost about twice that amount. An additional $200,000 would have been money well spent to enable communal leaders to make decisions based on accurate data. The federation’s mission is to service the Jewish community. In this case it has provided a great disservice, by allocating an expenditure without thinking through the implications of underfunded research.
None of the researchers who have directed previously conducted large Jewish federation major surveys would have agreed to direct a survey with a self-selected sample.
I hope that Jewish leaders who will consider using the new survey results to help their planning and decision-making will view the new study results most critically, and reconsider their use of the new data.
THE FEDERATION RESPONDS
The Jewish Federation of Northern New Jersey is sorry that Mr. Schwartz is so displeased with our market survey methodology, but we’re willing to agree to disagree with Mr. Schwartz on that.
However, we reject his implication that we were not transparent. As the Standard article made clear, we had no intention of conducting a purely demographic study and did not set out to do so, nor did we ever claim to. We were more interested in an actionable market survey of those who regard themselves as affiliated with the Jewish community, which would reveal their needs and desires and how well they are being served now. They are the most likely future users of Jewish services in our community. To that end, the survey has provided significant insights that will be beneficial to the entire community.
Jason Shames, CEO
Zvi S. Marans, M.D., President
Jewish Federation of Northern New Jersey
comments