[MUD-Dev] Role-Playing Games Are Not Dead

[Ling Lo]

This was posted a while back:

  http://groups.google.com/groups?selm=dwendt-0603001601410001%40statmac63.mps.ohio-state.edu&output=gplain

---<cut>---
From: dwendt at consultant.com (David Wendt)
Subject: A Response To The Adventure Game Industry Markey Research Summary
Newsgroups: rec.games.frp.industry
Date: 2000/03/06


From:  David Wendt, Ph.D.
           Professor of Statistics
           Statistical Consultant

Permissions: This file is based on data presented in the Adventure
Game Industry Market Research Summary.  The file in question is
Copyright 2000, Wizards of the Coast.  The file in question had
permission granted to be freely redistributed in or quoted in whole
or in part, provided that this attribution remained intact.

With regard to Mr. Dancey's Market Research Summary:

While I applaud Wizards of the Coast, and Mr. Dancey in particular,
for their openness with their market data, I feel that must express
some concerns with the manner in which they presented their
information.  While Market Research methodologies are commonly
accepted for internal corporate use, one must be careful when
presenting them in a public forum.  In this document, I plan to
address my concerns with this particular summary one at a time.  I
encourage my colleagues to comment on my discussion and in
particular on a number of open issues I point out in the text.

With regard to the 'Pokemon Effect': While I respect the decision
not present information on the trading card game (henceforth
referred to as TCG) component of the data, your survey was intended
to "determine information about [TCGs], [RPGs] and [MWGs]".  This
sends up an immediate flag with respect to my training.  Any survey
or test is set up to gather specific information.  When one changes
the use of the data, one is in danger of invalidating the
information.  Thus, we must already consider the rest of your
conclusions with a grain of salt.

My second major concern is related to your decision to exclude
population based on age.  Your acknowledgement of the impact of this
exclusion was a good first step.  However, a great many of your
conclusions are age- and time- based.  As a result, you have
potentially introduced a great deal of bias to your data.  This is a
second reason to be wary of the conclusions presented in your
document.

Additionally, you claim that the survey "accurately represents the
US population as a whole".  I will assume this is a misstatement.
The US population includes a large population in the over 35 group.
You might argue that your survey accurately represents the 12 to 35
age group in the United States.  This is also possibly incorrect,
however, as the screener and follow-up survey may have produced
further bias based on the information provided to the respondents.
Were the surveys identified as being sponsored by WOTC or some
related organization?  In addition, non-compliance also tends to
bias such surveys (who is the sort of person who would take the time
to fill them out?).  This non-compliance bias is well documented and
is part of the recent Census versus Sampling debates, for those who
wish to read more on this issue.

I will address further concerns related to the relationship between
age and time further later in this document.

Let us consider your conclusions in Section3: Basic Demographics.
You either have been careless in your explanation of your
conclusions, or have omitted some of the results you base those
conclusions on.  While you provide single level crossover values,
you do not provide the probability of multiple crossovers (defined
as, for instance, a TRPG gamer plays two or more of the other
categories.)  Comparing the crossover values you did provide to the
percent of women in gaming (which you call significant), the values
seem comparable.  This would imply that by your definitions, these
crossover percentages are significant.  (As an aside, 'significant'
has a very specific statistical meaning, so one should be careful
when using it in a public document.)

Otherwise, I am a bit concerned with the conclusion you draw.  I am
not at all convinced that the fact there is little crossover means
that extending a brand can be successful.  I would imagine there are
psychological factors to be accounted for, and would welcome the
comments of anyone trained in such matters on this issue.  Further,
I suspect there may be gender buying trends that you have not
accounted for when you say that "females, as a group, spend less
than males".  Again, I choose not to comment further, but encourage
those with appropriate training to do so.

Finally, for this section, while your conclusion that gaming is an
adult hobby is well supported by your data, I remind the reader
again that a significant portion of the populace has not been
included.  I suspect that the percentages of gamers in the lower age
brackets has been unduly inflated as a result.  Further, your final
sentence in this section, "...the existing group of players is aging
and not being refreshed by younger players at the same rate as in
previous years" is completely unfounded.  First, without previous
years market data, we have no benchmark to compare these number to.
Secondly, later in your own document, you observe that gamers join
the hobby at various stages of life.  This will directly increase
percentages as age increases by definition.

In sections 4 and 5, I again encourage the reader to be aware that
all of these percentages as biased by the exclusion of respondents
over 35.  In particular, levels of education and marital status have
direct relationships to age.  A 12 year old is much less likely to
have done post-graduate work or to be partnered than a 35 year old.
This must be considered when reviewing the results.

I have additional concerns about the results presented in section 5.
First, I must revisit my concerns about potential bias due to the
attributions of the survey.  If response were biased by perceived
ownership (that is if the respondent believed he was responding a
specific company), then all the percentages are in doubt, but in
particular those about "past game experiences".  Different game
companies have historically had different attitudes about a number
of the experiences the respondents were queried on. This should be
kept in mind when considering the results.

Secondly, the amount of spending is likely correlated to age.  Due
to the impact of education and employment, older gamers are likely
to have more expendable income.  Thus increasing their total
spending.  Further, the longer a particular gamer has been in the
hobby, the older they are, thus taking us back to the earlier
argument.  As a result, subdividing spending by age is of minimal
information, and likewise with the D&D specific spending.

Regarding your monthly spending comments, I will simply encourage
you to look to hypothesis testing.  I suspect that the perceived
differences are not significant.  That is, due to variations in the
data, there is not sufficient evidence to say that there are
spending differences.  Such tests are well documented in basic
statistics texts, and as a result, I will comment no further on
them.

I do not find the results on the spending of current versus lapsed
players overly interesting.  I suggest an alternate measure to you.
That is the Mean Total non-D&D Spending per non-D&D game played.
Based on your numbers, Mean Total non-D&D spending is ($1273 - $895
=) $378 for current players and $1068 for lapsed players.  Dividing
these by the Number of non-D&D games played (1.2 and 2.3
respectively), gives the following:

        Mean Total non-D&D Spending
                per non-D&D RPG Played:  $315 / $464

These numbers are much closer indicating that perhaps there is
little to no difference in non-D&D playing.  It also suggests that
perhaps RPGs other than D&D represent less of an investment for
players.  I also suggest that there may be a bias inflicted by the
age truncation that may explain the difference between current and
lapses spending.  Further, lapsed players may be overestimating
their spending and current players underestimating theirs.  This is
a psychological factor that I would encourage professional or
academic comment on.

I am a little concerned about your percentages regarding "what games
TRPG players play" and "retail type".  First, does the 35 or older
truncation bias these results?  Do any of the other potential
sources of bias discussed in this document affect these numbers?
These are points for consideration.  Relating more specifically to
what games are being played, the choice of games over companies is
disturbing and introduces a newer source for error.  I suggest that
for future surveys, this sort of question be replaced by "Do you
play TRPGs from the following companies?  Wizards of the Coast,
White Wolf, ..., Other, Unknown."  A number of solid games were
excluded from your list and not all companies were included equally.
As a result, this data, as it stands, is useful for little more than
trivia (ignoring any bias issues).  Percentages for games from the
same company cannot be added with any meaning, due to overlap, nor
can this be used for any measure of marketshare.

I am curious about how you moved from "on a scale from 1 to 5" to
percentages in the question where players where asked to describe
aspects of their games.  While the data points are potentially of
great interest with respect to game design, without more information
on how the data was gathered and the results computed, I believe
this section tells us very little.

In conclusion, I again applaud Wizards of the Coast for their
openness with their market data.  I encourage other members of the
gaming community to be equally forthcoming.  However, due to
decisions made in the data collection phase and in the results
development phase, I caution anyone who reads this survey to be
cautious in drawing any conclusions.  There is not only a danger of
significant bias, but important correlations in the data have been
seemingly ignored.  Market research is a powerful internal corporate
tool, but must be taken as such when released to the public.

David Wendt, Ph.D.
---<cut>---

--
  |   Ling Lo
_O_O_ ling at slimy.com

_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev