Posting Rules in Online Discussions Prevents Problems & Increases Participation

Rule postings made newcomer comments 7.3 percentage points more likely to follow the rules, increasing newcomer participation by 38.1% on average, in a field experiment of 2,214 discussions on r/science.

by J. Nathan Matias, October 8, 2016

Flowchart illustrating the effect of a sticky comment on the choices made by a newcomer. Upon seeing the sticky, a newcomer might choose to post within the rules, or they might decline to post, perhaps because they see their comment isn't allowed, or because they think it's not the place for them, or because it's too much work to follow the norms in the community.

r/science, like many subreddit communities on reddit, has rules for the kinds of comments they allow. Does posting the rules actually prevent people, especially newcomers, from commenting outside those rules? Might there be a side effect on newcomer participation?

As a community with over 13 million subscribers, over 1200 moderators, and thousands of moderation actions per week, even a small effect could make a big difference. Across 29 days in September, I worked with the moderators of r/science to test this question with CivilServant. In this A/B test, we posted sticky comments to some threads and not to others.

Sticking a rule comment to the top of discussion threads increased a newcomer's probability of posting a first comment within the rules, from a fitted chance of 75.2% to a fitted chance of 82.4% on average within r/science

We found that adding a sticky comment with the rules has a positive 7.3 percentage point effect on the chance that a newcomer’s comment will not be removed by moderators on average across r/science, holding all else constant. And rather than reducing participation, posting the rules increases the incidence rate of newcomer comments by 38.1% on average. In the experiment, newcomers are accounts making their first r/science comment in the last 6 months. The 20,385 newcomer comments were 29.7% of all comments in this period.

But there’s a catch! We found that sticky comments had opposite effects in non-AMA posts compared to AMAs (live Q&A conversations with prominent scientists). Posting the rules to a non-AMA thread caused a 59% increase in the incidence rate of newcomer comments, but in AMA threads, sticky comments caused a 65.5% decrease on average, the opposite outcome. The difference is illustrated here:

The Overall Effect of Posting the Rules

Beyond newcomers, sticky comments increased rule-compliance across all commenters by 2.2 percentage points, on average across all posts. Sticky comments also increased the amount of moderation work per post. Although they reduced the chance of an individual comment being removed, they also increased the number of comments. So overall, posting a sticky comment increased the incidence rate of all comment removals by 36.1% in non-AMA posts, and decreased the incidence rate by 28.6% in AMA posts, on average across r/science.

Sticking a rule comment to the top of discussion threads increased any comment's fitted chance of being within the rules from 88.3% to 90.5% on average within r/science

That makes decisions about sticky comments a tricky one: on one hand, they are a powerful tool for improving participation and growing a community. On the other hand, the result might be more work for moderators overall.

How Can We Make Sense of These Results?

Theories from social psychology predict that sticky comments affect social norms, people’s beliefs about what others consider acceptable. In particular, rule listings constitute what Robert Cialdini would call "injunctive norms,"" rules that "specify what ought to be done… these norms enjoin it through the promise of social sanctions”— the threat of comment deletion [6]. Theories of social norms guide our first hypothesis, that posting the rules might increase the chance that newcomer comments will comply with the rules. Elsewhere, experiments in applied social psychology have found similar effects of posting signs on littering behavior [1, 2], smoking in hotel rooms [3], environmental conservation by hotel guests [4], and crime reporting [5]. So we had good reason to think that this might work.

In my initial reading, I discovered that social theories disagreed on the effect of sticky comments on the overall levels of participation. In one view, people who see the rules might choose not to participate. Perhaps seeing the rules might convince newcomers that their comment isn’t acceptable, so they stay out entirely. Rules also increase the work of commenting and perhaps reduce participation [7]. Finally, newcomers encounter rules during their “investigation phase” of a community. After seeing the rules, someone might decide that they don’t fit and may never participate [8].

Theories of newcomer socialization from 'Group Socialization and Newcomer Innovation' in the Blackwell handbook of social psychology. The chart shows several stages in the engagement of an individual with a group, from Investigation, Socialization, Maintenance, Resocialization, to Remembrance. The 'Investigation' phase is highlighted.

On the other hand, if newcomers see that a community has policies they like and believe that it is well-moderated, they might be more likely to participate, as one lab study of news comments found [11].

By running an experiment, we were asking a practical question about how to moderate. We were also trying to add evidence to this disagreement among different theories of social behavior.

What Don't We Know?

One experiment never settles a question, especially at a time when results have sometimes been difficult to replicate. Here are questions that remain open:

  • Do these effects apply beyond r/science to other communities?
  • Why do we see the opposite effect between Q&A sessions and more ordinary discussion threads?
  • Are there things about the wording of the comment that influence its effectiveness?
  • Is the sticky comment more effective for some kinds of rules than others?

One way to answer these questions is for more subreddits to try similar experiments. If you are interested, you can Contact me on reddit to discuss running a similar experiment and sign up for email updates.

Learn More About This Experiment

My PhD involves supporting subreddits to test the effects of their own moderation practices. Public reddit comments from our conversations about the experiment may make it into my dissertation. In any publications, comments are anonymized and obscured to prevent re-identification.

I designed this experiment together with r/science moderators, and it was approved by the MIT Committee on the Use of Humans as Experimental Subjects. If you have any questions or concerns, please contact natematias on redditmail.

This experiment, like all my research on reddit to date, was conducted independently of the reddit platform, who had no role in the planning or the design of the experiment. The experiment has not yet been peer reviewed. All results from CivilServant are posted publicly back to the communities involved as soon as results are ready, with academic publications coming later.

Full details of the experiment were published in advance in a pre-analysis plan at osf.io/knb48/. Notice that we hid the sticky comments from moderators during the experiment to reduce the influence of the sticky comments on their decisions. If you are interested in the statistics, I published full details of the analysis to github.

References

[1] Susan M. Reiter and William Samuel. Littering as a Function of Prior Litter and The Presence or Absence of Prohibitive Signs. Journal of Applied Social Psychology, 10:45–55, 1980.

[2] Yvonne AW De Kort, L. Teddy McCalley, and Cees JH Midden. Persuasive trash cans: Activation of littering norms by design. Environment and Behavior, 2008.

[3] Harold H. Dawley, John Morrison, and Sudie Carrol. The Effect of Differently Worded No- Smoking Signs on Smoking Behavior. International Journal of the Addictions, 16(8):1467– 1471, January 1981.

[4] Noah J. Goldstein, Robert B. Cialdini, and Vladas Griskevicius. A Room with a Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels. Journal of Consumer Research, 35(3):472–482, October 2008.

[5] Leonard Bickman and Susan K. Green. Situational Cues and Crime Reporting: Do Signs Make a Difference?1. Journal of Applied Social Psychology, 7:1–18, March 1977.

[6] Robert B. Cialdini, Carl A. Kallgren, and Raymond R. Reno. A focus theory of normative conduct: A theoretical refinement and reevaluation of the role of norms in human behavior. Advances in experimental social psychology, 24(20):1–243, 1991.

[7] Carsten Eickhoff and Arjen de Vries. How crowdsourcable is your task. In Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM interna- tional conference on web search and data mining (WSDM), pages 11–14, 2011.

[8] John M. Levine, Richard L. Moreland, and Hoon-Seok Choi. Group socialization and new- comer innovation. Blackwell handbook of social psychology: Group processes, 3:86–106, 2001.

[9] Alan S. Gerber and Donald P. Green. Field experiments: Design, analysis, and interpretation. WW Norton, 2012.

[10] J. Scott Long. Regression models for categorical and limited dependent variables. Advanced Quantitative Techniques in the Social Sciences, 7, 1997.

[11] Wise, K., Hamman, B., & Thorson, K. (2006). Moderation, response rate, and message interactivity: Features of online communities and their effects on intent to participate. Journal of Computer‐Mediated Communication, 12(1), 24-41.