At Gen Con 50, I spoke on a panel with Jake Given, Carly McGinnis, and Matt Fantastic that focused on different aspects of playtesting. So, here’s a post about data tracking when playtesting your games, which is only a portion of the panel, but is that part that I felt most knowledgeable about. This applies to both tabletop and digital games.
When it comes to asking questions before, during, and after your test, go into your test with specific questions in mind. It’s best if they’re already written down, and you should do this before your test starts and when anything comes up during the test. It’s up to you if you want your testers to know what you’re looking for. This may affect your data, but it can also encourage them to take certain risks they may otherwise not do if they were simply playing as normal. For example, when I’m a player in my own tests of Apocrypha, I almost always mutate a confrontation if I can. In normal play, I am too afraid of the potentially bad consequences that I don’t do it unless I feel like I’m seriously in a bad spot. I know that our target players are more likely to mutate than I am, so I just take the risk to simulate being that kind of player.
Your questions need to also be asked in a way that will give you the most data from a player (who is most likely not a game designer). Ask open-ended questions! A lot of these questions will assume you suspect the issues with your game already, and that’s okay. When you’re gauging feeling and fun, the question “did you have fun?” doesn’t give you the answer you’re actually looking for. Try “what parts of the game did you most enjoy?” or even “how did it feel to kill the dragon with your sling-shot?” A negative question like “was the game too long?” will also not give you a usable answer. Watch for when players take out their phones or their eyes gloss over. At what points do they go to the bathroom or get a snack without being prompted? You usually don’t even need to ask a question about this, but some players are too nice and will sit through anything. A question like “which parts of the game could be cut or didn’t seem to make a difference in play?” will help.
There is also the issue of when to ask questions. I prefer to ask at the beginning and end of the test and if I have a quick question about something during gameplay, I keep it short and have a larger discussion after the game is over. Sometimes, what you’re really trying to test doesn’t come up until mid-game. There is no reason to start your test at the beginning. A lot of new designers are afraid of “ruining” their data by missing something important. You won’t. It’s okay. You can test an entire game later. Simulate a specific game state in a few different ways: where players are the most prepared, half prepared, and under prepared. The more random your game is, the more difficult this might be, but it will still inform you about that game state.
Be aware of how many times you have tested a specific part of your game and the way you tested it. If it’s only been a few times and it’s not working, do you need new materials or different players? Do you need a new environment? Do you need more time? Usually you can tell what’s up with a mechanic after a few tests. If you’ve tested something a lot and it’s still not working out, it may be time to remove that mechanic or change it more dramatically.
There is lots of hard data you can record without asking questions. Data you should always record includes your testers full name and contact information (usually email), and gameplay length (not including test discussion time, but including any discussion about setup or strategy). Data that is relevant to your game is going to be specific, so here are some examples of what I record when testing:
- Apocrypha Adventure Card Game:
- the number of cards left in the timer deck at the end of the game
- which virtues players chose to use (or had no choice but) when confronting
- characters played and who played them
- Betrayal at House on the Hill: Widow’s Walk:
- number of haunt rolls until the haunt was triggered
- the number of tiles are on each floor at the end of the game
- How long did haunt setup take from the heroes’ side and the traitors side?
This data is the kind that gives you a frame of reference for all your notes and is evidence for any issues in the game. So, in your game it might be “how often are the players using their equipment”, “how many times does a player help another player (co-op) and what were the results of those assistances”, and “which class won the most often and which lost the most often?” This is data you do not need to ask players about.
You will also want to record what players say. Use quotes as often as you can and record who said what, and refrain from referring to your testers as “player 1” and “player 2.” Attach their name to their experience; this way you have the context of who that person is and what kind of player they are, and their feedback is easier to contextualize and apply to your game. Your players may suggest how to fix a problem, and while they are almost never right about how to fix it, the way they feel matters more. Your game can be the most numerically balanced game in history, but that doesn’t matter if no one feels good playing it (if fact, most too-well-balanced games feel pretty bad). As a game designer, it’s up to you to figure out the best solution.
To wrap up this long ass post, here are the biggest data tracking mistakes that I see:
- Not playtesting the game’s final rulebook; do not say anything as players attempt to set up, play through, and finish your game without any input from you. If a player asks a question, give them a non-answer like “I’m not sure, what do you think it should be?” You won’t be in the box with the players.
- Going in without questions or not knowing what to test – “winging it.” You may get some data, but not as much as you could have otherwise and it can make your testers feel like you wasted their time which is a big no-no if you want them to test with you again.
- When you’re wrapping up a project, winging the test is a little more acceptable because it’s really just to catch those little problems that have not otherwise appeared.
- Not getting your tester’s contact information. You may need to ask them questions later or come test again to compare their previous experience of the game.
- Thinking the data you received from a test is useless. It’s not. Even if it’s not data you wanted, it still tells you something about your game and how a specific player type may respond to it. Do you want this player to be part of your target market? Do you need to make sure they know they may not enjoy this game.
I hope this helps you get better data from your tests! Happy testing 🙂