I recently conducted a sociology experiment called the "Wallet Test"... I deliberately dropped 100 wallets in front of hidden cameras to test honesty. You can read the details on my website here: http://www.wallettest.com/

I've received many complaints from people saying that my experiment "...was not at all scientific." So could someone please tell me what qualifies as "Scientific" and how I could have made my test more so?

Thank you in advance for your help.
- Paul Kinsella (A.K.A. "Suncrafter")

Hi Suncrafter. Welcome to AnswerPool.

In order to find if something is scientific you must determine a cause and effect based on variables. This is known as a scientific method. You have to determine what cirmumstances would influence the outcome. This would be done by dropping the wallet in two different areas of the neighborhood, two different times of the day, two different amounts in the wallet etc... You can only make a hypothesis if there is a cause and effect between the variables and a controled situation. I did see that you dropped the wallet in various locations but there were no real notes taken. You observed race, assumed age and location. In some cases a young girl dropped the wallet and her grandmother turned it in. You don't know if the girl would have kept the wallet. An older woman at the post office returns the wallet. A student at the University keeps the wallet. This isn't very scientific. You were trying to find out if people were honest right? I think you need to be more specific. You would have to drop 100 wallets in one location to get a control. You then have to base the variables on how that effects the results. What you must do is change something in each case and place to see if your results change. I think a more scientific experiment would be to drop a wallet with various amounts at the same location. Are people who shop at one particular grocery store more honest than those who shop at another? Drop the wallet 100 times with $50.00 in it at one store. Drop the wallet 100 times with $50.00 in it at another store. Vary the amounts and times of the day. You then would calculate your raw data of dropping the wallet compared to the amount and the store you dropped it at. It has been a long time since I conducted any experiments and I'm basing this on what I remember. I do think your experiment was very interesting. I see you put a lot of hard work into it and I am very impressed with your site. I think you need to just fine tune your experiments and it will be more scientific.

Thank you. That helped.
I also recived a helpful link from someone else on another forum - http://teacher.pas.rochester.edu/phy_labs/AppendixE/AppendixE.html

Thank you again for your help,
- Suncrafter

Well, since you asked for a critique...

My main concern would be the lack of a statistical analysis. I notice that you excluded "old black people" due to only having 1 member of that group. That was good. However, you did consider a group of only 9 ("black males"). My back-of-the-envelope calculation suggests that there is no statistically significant difference between this group and "white males" or "black females."* In fact, most of the differences between groups do not look statistically significant to me. For those that do, there is the problem of separating variables. Again, separating the people by more than one characteristic was good, but you did not go far enough. Your results (although, as I said, they mostly don't look statistically significant) suggest that race, age, and gender all have an impact. Separating by only two of these variables at a time still leaves room for comparisons between samples biased in the third variable. You could break the results down further, sorting by all three variables at once, but you'd have even smaller sample sizes and even less statistical significance. You could also use multiple regressions, but I'm sure you'd have a similarly low statistical significance. To make this experiment good, the main things you need are a much larger sample size (so that you can assure that your subsamples have large sample sizes) and a statistical analysis of the results.

Other concerns are as Claire noted - controlling the untested variables. Behavioral research is difficult to do well because there are so many variables which the experimenter can't control and may not even be aware of.


-------------------------------------

*Back of the envelope calculation for "black males" gives +/- 33% at the 95% confidence interval, which means that there's a 95% chance that the true "honest" percentage lies somewhere between 11% and 77% (and a 1 in 20 chance that it's outside of that range). The other groups will have a narrower range due to larger sample sizes, but there is certainly no statistical difference between "black females," "white males," and "black males." Even the "white females" / "black males" comparison looks to me like it is on the edge of statistical significance. You'd have to do the calculations to determine on which side of the divide it lies (keeping in mind that the 95% standard is somewhat arbitrary and, by definition, is wrong 1 in 20 times). All that, of course, assumes that random sampling is the only effect and that other variables (age or location, for example) are not biasing the results of a particular comparison (as discussed earlier).



You'll need to drop a lot more wallets to get meaningful results.

UPDATE:
Recently they talked about WalletTest.com on "Penn Jillette Radio Show", CBS Radio! That is super-cool because I'm a big fan of Penn Jillette! You can listen to it here:
http://www.pennfans.net/view/Audio_Archive/PennRadio/Pe...dio.Show.2007.01.29/

That was very cool!

This would be a hard test to control. In one I watched, for example, the man (who returned the wallet) appeared to be with a young man I presume is his son. Did the boy's presence compel him to model good behavior, or would he have returned it, anyway? We will never know.

The controls, variables and ethics of conducting experiments become quite complex in social sciences. Regardless of scientific accuracy I think this was an interesting and fun study in human behavior.