Lessons Learnt On What To Ask On a Blind Play Tester Feedback Form

What did I learn about blind play testing questions from my recent blind playtest of my game Post Position?  Back in October I came up with a series of questions to help me to construct a blind play tester feedback form.  (Link to: What to Ask on a Blind Play Tester Feedback Form?)  The feedback forms are now back and I will give a quick run down over the questions that worked well and the one that did not.

 

Did you like the game, why or why not?

 

Here is a question that was not on the original list of questions to try.  The question was a standard question on the old UNPUB feedback forms.  I discovered for a blind play testing this question was wasted ink.  The problem with the question was that people are way too nice.  

No one ever said they disliked the game, even though some players found the game to be difficult and too long.  Plus answers were short on detail so it was not clear what they really liked or disliked about the game.

 

That said, the sort of questions that did provide useful details about the parts of the game play testers enjoyed and the parts of the game they disliked was when I asked the straight up “What was your favorite part of this game?” and “What was your least favorite part of this game? “  Players were liberated from trying to come up with a reason why they liked the game or disliked the game and just went straight to the details about the parts of the game that were good and bad.

 

What I discovered was the most important question was actually a group of questions that I will call the four essential questions.

 

What was your least favorite part of the game?

 

Name one improvement you would make to the game?

 

What strategies did not or did work for you in the game?

 

What was the hardest rule to learn and why?

 

These four essential questions not only highlighted which rule in the game was the hardest for players to learn, but they also provided illustrations on how that rule affected overall game play.  In form after form, when a player listed their least favorite part of the game they would look to find an improvement to fix that part.  When they talked about their game strategies you could observe how that mechanism made it harder for them to succeed.  And when they talk about the hardest rule to learn it almost always was the player least favorite part of the game.

 

The four essential questions combined gave me an insight into not only what rule gave the player a problem, but how that problem affected their gameplay and what sort of things I need to try to fix that problem.  

 

For example, short position selling in my game Post Position could be a player’s least favorite part of the game.  Then the player would suggest a change to the short position selling form as an improvement to the game.  When the player talks about game strategy they then talk about how short position selling was a bad strategy for them and that there were uncertain why a player would ever engage in short position sales.  Finally, when asked what was the hardest rule to learn the player listed short position selling.

 

What did I learn? I learned the player did not understand the purpose of short position selling and was therefore not able to execute the maneuver in the game.  This means I need to do a better job of not just explaining how to conduct a short position sale but also include a short explanation on why a player might want to engage in a short position sale so they have a little intuition when it should be used.

 

This is concrete feedback that I will use to improve the game.  No matter the issue with the game, the four essential questions produced similar feedback.  These are the questions I am convinced should be in ever blind game play testing feedback form.

Random Lessons in Prototype Shipping

If you don't read the rest of the post and are getting ready to mail out a prototype for blind playtesting remember these two lessons:

 

1. Include your contact information in many different places with your prototype.

 

2. Americans, ship your prototype using a U.S. Post Priority Mail Flat Rate box.

 

Of course, to find out why, you need to keep reading.

 

Last October my board game Post Position left the safety of my home to travel and visit others for a series of blind play tests.  To keep prototype costs low, I was circulating a single prototype between different playtesting groups.  For this to work it needed to be easy for the host play tester to receive the package and to forwarded it on to the next destination.

 

My solution was to package the game in a flat rate size medium #2 United States Postal System Priority Mail box.  In the box I included pre-print labels with paid postage for each destination.  That way the host play tester only had to tape the next label onto a new free U.S. Priority Mail flat rate box and drop it off at the local Post Office. 

 

With all the pre-printed labels and an instruction sheet packed together I was on top of the world.  I didn't even have to worry if the postage was off due to a random weight change.  I thought to myself I was prepared for anything that could go wrong.

 

That was until I received a frantic message from the first play tester.  Post Position apparently wanted to make a side trip to Georgia.  The game hypnotized the host into thinking it was a t-shirt and not a board game.  So under a hypnotic trance the host placed a pre-paid shipping label onto the Post Position box to a t-shirt company in Georgia.

 

At this point we convened a crisis committee.  The host quickly contacted the t-shirt company and arrange to get the prototype mailed back to him.  Weeks past and still no game had returned from Georgia. 

 

I was prepared to write the prototype off as permanently missing in action when a human resources manager at Alpha company sent me a message saying they had Post Position.  Something had gone wrong with mailing Post Position back to the host and the U.S. Postal Service returned it back to the t-shirt company.

 

That is how I discovered the best things I did putting together the prototype for blind play testing was to include a lot of documents with my e-mail on it.  The t-shirt company found my e-mail and reached out to me.  

 

After thanking them profusely for contacting me and promising to say lots of nice things about them (you should all buy one of the t-shirt brands from alphashirt.com) I learned the other benefit of mailing the prototype using flat rate U.S. Priority Mail boxes with pre-paid labels.  I was able to e-mail a pdf copy of the pre-paid USPS shipping label to the t-shirt company.  Since the rate was a flat rate from any destination I had no worry about the package having the wrong postage.  And, if the Post Office had trouble delivering the box to its next destination, I would get the prototype game back since the return address was to me.

 

Now I eagerly await to see what sort of hypnotizes it performs next.  If only I could train it to hypnotize publishers to pick the game up instantly.

What To Ask On a Blind Play Tester Feedback Form?

My game Post Position is traveling across the country so playtesters can test the written rules for the game.  Players will be participating in a blind playtest - learning and playing the game with just the rulebook.  Usually after a game I get feedback on that specific game through conversation with other players, and some written reviews.  Without being present to observe, I need a feedback form that lets me know how well the group understood and implemented the rules. 

To help construct a good feedback form I am breaking down some of the questions on my form and explaining why I think I should ask them.  The word think is underlined because I do not know yet how well my form will work. I will write a follow-up blog later this year on how well each question worked.

There are many good feedback forms other folks have already put together.  UNPUB has a very good standard form they use at all of their game testing events.  The challenge is the UNPUB form is used at UNPUB events where the designer is personally teaching the rules to the player...aka not a blind play test. 

Also, the UNPUB form is used in situations where you can watch the play testers play the game.  So I need a feedback form that really tries to capture what actually took place within the game. These are the questions I think will tell me if the rules are working. 

Feedback From Questions

Were you able to play the game?

Silly? I think not.  The whole purpose of sending the game out to people who have never played the game before is to learn if they can play the game using just the rule book.  You pray the answer to the first question is some sort of yes, otherwise you can stop reading the survey at this point and start burning your current rule book so you can start over.

How long did it take you to learn the game?

Just because someone learned the game did not mean they learned it in a fast enough time for other people to enjoy it.  For example, I love the game Diplomacy, but I spent over 4 hours learning how to play the game with my high school friends.  If we were not dedicated board gamers then there was no way we would have the patience to spend so much time to learn to play the game.  (I know, you are rolling your eyes because only dedicated board gamers know of the game, yet alone consider playing it).

In your own words summarize how to play Post Position?

"Wait," you say, "didn't you already ask them if they learned to play the game?  Why do you need to ask them to summarize how to play the game?"  Just because they think they learned to play the game does not mean they actually learned to play the game I designed.  I am not there to see them play it, so I need to find questions that will reveal to me how they actually played the game.

What was your mood before learning the rules?  What was your mood after learning the rules?

Yes, I am asking questions about emotional stuff.  Why, because I want to gauge how emotional wearing it was to learn the rules.  Rules not only exist to make sure everyone keeps in line when they play the game, rules exist to entice people to want to play the game.  If my rules drains the fun out of the room there is a decent chance that the game goes back into the box before a single horse moves along the race track. 

What was the hardest rule to learn and why?

Notice I assume that a rule was hard to learn.  This will bias their answer to find a rule that was hard.  I am ok with this because no matter what they will identify the most frustrating rule explanation in the rule set.  Odds are improving that one rule will by itself make people much happier with your rule set.

Did the graphics make it easier to learn the rules?  Please explain why or why not.

Post Position includes some graphics to help explain the game.  That said, we need to know if the graphic made it easier to learn to play or just created more confusion.  I ask why or why not because it creates another opportunity to let me know if they created their own house variant of the game or if they actually learn to play the game I intended to teach them with the parchment in the game box.

Did you understand the rules better after finishing your first game?

Some people learn by doing versus reading.  Everyone has a friend who never reads the rules and relies on someone who knows the game to teach them.  I want to find out if those who could read the rules were able to teach the other players how to play by playing the game with them.

How comfortable would you feel teaching someone else the game, why?

Confidence in your ability to teach the game is a measurement in how much you think you know about the game.  I ask why because it tells me which part of the game they are the least comfortable with.

Describe what strategies worked or didn't work for you in the game?

I find it a good idea to ask the same question in a different form.  Details may come out that helps clarify players answers on previous questions.  By describing in this question what strategy did or did not work for the player they are also describing the rules of the game they used when they played.

 Conclusion

So are these the only set of questions that you should be asking?  No, you should also include a number of the standard questions found on the UNPUB game review form.  Do you like the game?  What was your least favorite part of this game? etc.

And now we wait.  You wait to see if my forms succeeded to illicit useful feedback. I wait to see if Post Position's rulebook needs minor tweaks or a trip to my fireplace.