Aiming for an unbiased assessment of the BRC population

Summary of the challenges and implemented solutions

Written by  Dominic “Hunter” Beaulieu-Prévost

Edited by David “Scribble” Nelson-Gal

May 3rd, 2015

For many years prior to 2012, the Black Rock City (BRC) Census was actually a convenience sample. Specifically, the data reflected only the percentage of the population that chose to fill out and submit the paper surveys placed at numerous kiosks throughout BRC. For example, though we know that women make up between 40%-42% of the population historically, prior to 2012, the BRC Census reported majorities of women in some years simply because women were more likely to fill out the survey. These self-selection biases are also prevalent in a broad variety of demographic tendencies.

To correct for these errors, starting in 2012, the BRC Census started doing a random sampling of the vehicles arriving at Gate to get a relatively unbiased estimate of the BRC population. This random sampling could then be used as way to adjust the survey data so that it would more accurately reflect the population.

The random sampling procedures have worked well and we are quite confident that the estimates derived from it are good, at least for the covered populations. However, because of certain obvious uncovered populations, measures like the percentage of Virgins in the population have been over estimated. Population coverage is our main challenge and we have been improving the method every year to increase coverage.

Adjusted weight compared to earlier reported numbers for Virgins

Adjusted weight compared to earlier reported numbers for Virgins

The main issue

  • In a nutshell, the random sampling provides a valid estimation of the sociodemographics for the population of BRC citizens who theoretically had a chance of being randomly sampled. Primarily, the uncovered populations are populations that arrive either before or after our sampling windows as well as populations that arrive through a ingress point not covered by our sample (e.g., the BRC airport).
  • For the uncovered populations, the best that we can do is to infer the sociodemographics from those of the covered population. This inference can be done statistically, but it is less reliable and could be considered as a type of « informed guessing ».
  • Up till recently, our main solution has been to increase the coverage of the random sampling as best as we could.
  • Additionally, being able to access more precise daily population numbers from the Burning Man Org, allowed us to create more sound weights associated with arriving populations.
Calculated English Speakers using newer weighting procedure vs historic reported numbers

Calculated English Speakers using newer weighting procedure vs historic reported numbers

Yearly population coverage

In 2012 we sampled Gate entries from opening (Sunday) to Wednesday

Uncovered population

  • Early arrivals until Saturday
  • Late arrivals beyond Wednesday
  • Airport arrivals
  • Shuttles (they were officially included, but often refused to participate)

In 2013 we sampled Gate entries from Friday (pre-event) to Wednesday

Uncovered population

  • Early arrivals until Thursday
  • Late arrivals beyond Wednesday
  • Airport arrivals
  • Shuttles (they were officially included, but often refused to participate)

In 2014 we sampled Gate entries from Friday (pre-event) to Wednesday and Burner Express Bus (BxB) (who used a dedicated lane starting in 2014). Rather than a random sampling of vehicle occupants as we do at Gate, each BxB rider was handed a sample survey at the Gerlach transfer station and was asked to fill out the sampler forms which were then collected at the Bus Stop in Black Rock City. Spot checks of arrivers noted near universal compliance but not every rider completed and submitted their forms to the available Bus Stop kiosks. Conservatively, we estimated compliance around 60%.

Adjusted vs Historic calculation for Males

Adjusted vs Historic calculation for Males

Uncovered population

  • Early arrivals until Thursday
  • Late arrivals beyond Wednesday
  • Airport arrivals

Priorities for 2015

  1. Pre-event coverage seems to be the most important issue
  2. Airport arrivals (being explored)

Note on airport sampling: Sampling the airport arrivals is difficult because arrival patterns are sparse, making stationing volunteer samplers somewhat unproductive. Moreover, unlike BxB, we don’t have a mechanism to hand sampler forms to arriving participants at some embarkation point to be then collected on arrival. Finally, the proportion of burners entering through the airport seems relatively small (although unknown to us).

This said, we know the demographics of these participants are probably different due to the cost involved. Survey respondents who indicate “Airport” as their entry point are asserting significantly higher incomes than the general population and may differ in other distinct ways that could be meaningful. Additionally,  there is a goal to increase the number of people who enter through the airport in order the lessen impacts on roads leading to the event. Therefore, coming up with a way to sample participants who enter through the airport has become more important in making sure our weighting of the online survey is as accurate as possible.

Improvement in the estimation method and impacts on the estimated sociodemographics

As mentioned earlier, population coverage of the random sampling increased between 2012 and 2014, which improved the reliability of the Census results. However, another important improvement was also made in 2014 on the calculation method. By discussing with the Burning Man Organization’s Communication department, the Census team was able to access more precise data concerning the daily population peaks during ingress between 2012 and 2014. This information was then used to recalibrate our estimation method.

By comparing our old calculation method to this new and improved method, we were able to assess the level of reduction in the biases. The main improvement was that we realized that the old method underestimated the proportion of early arrivals, which slightly biased most of the estimates. Hopefully, the differences between the estimates produced with the two methods were generally small. However, an important indicator was more substantially affected than the others: the proportion of virgins.

If you go back to the original Census reports for 2012 and 2013, you will see that the estimated proportion of virgins was estimated at approximately 39% for 2012 and 40% for 2013. As for our preliminary estimation in 2014, released during the event, it was at 41% of virgins. After improving the calculation method, we now estimate the proportion of virgins at approximately 35% for 2012, 39% for 2013 and 35% for 2014. The difference is not huge, but since the proportion of virgins can be a sensitive topic, any improvement in the estimates is appreciated.

Due to the changes in the calculation method, the estimates in the 2012 and 2013 reports cannot be directly compared to the improved 2014 estimates. Previous reports are not wrong, they are simply a bit less precise. However, to allow you to compare estimates from year to year, the 2014 Census report will include a new section: Trends analyses! You should thus expect in a few weeks blog posts presenting various trends from 2012 to 2014 based on our improved calculation method. We hope that you will enjoy this new addition to the Census report.

Editors note: Hunter is our primary statistician and this post explains the weighting process and how things have evolved.

Advertisements

One response

  1. Pingback: How Many Burns Have You Attended? | Black Rock City Census

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s