An Episode with Playtesting

We’ve released a lot of games on several different portals so we’re quite experienced with the Greater Internet Bastard Theory and how it is prevalent amongst what seems like a significant portion of the Flash game playing community. Last month we launched a single-level teaser of the upcoming Battalion games, and I must say, even though I usually can brush off this type of thing, many of the responses to the teaser level really left a bad taste in my mouth.
We wanted to put out Battalion: Skirmish for a few different reasons:
- We wanted to raise awareness of the fact that there were additional Battalion games in the works, that they were coming soon, and that there would be a level editor and competitive multiplayer.
- We wanted to recruit additional beta testers for Arena, the multiplayer version of the game.
- We wanted to pimp the iPhone release of our puzzle game, Robo Riot.
- We wanted to get feedback about all the changes we made since the first Battalion, Nemesis, including all the new graphics.
- We recoded significant portions of the game (including a total rewrite of the AI, the UI and many of the rendering systems), so we wanted to get this code battle tested.
We released the game early on a Friday evening. By the time I went to bed, the game was hovering around almost a thousand plays and had a very respectable rating of almost 4.2 on Kongregate.
When I woke up on Saturday morning, the number of plays had jumped to almost 20,000, and the score had dropped to almost 3.6. Although that is only a 12% drop, it represents, for me, the difference between a Great Game and a Decent Game. I was really surprised. Battalion is a GOOD game, how could it be at 3.6?
After my initial panic, I realized that my biggest mistake was in failing to manage expectations. In the description on Kongregate I clearly stated that this was just a teaser level, and what our goals with this release were. When I read over all the comments, it was clear that the length of the game was failing to meet the player’s expectations and so they felt compelled to punish me for it. They hadn’t read the game description. My first reaction was to ignore people who were going to be that stupid. Then after I settled down a little, I understood that I couldn’t expect people to read the game description when even I didn’t read them most of the time.
So, I put out a new build that made it very obvious that the game was just a teaser level and that it was a sneak peek (actually, I said sneak peak which I left up on purpose), and updated the description:
Some people missed in the description that the purpose of this game is to get some community feedback on the new features, and also to give people a sneak peak at the game itself. I updated the title screen to show this more clearly. I don’t know if its too late to change anything, but such is life. I apologize to anyone who was expecting anything more than just a mission showing off a couple new units and the new graphics.
I think it is pretty obvious from this little excerpt that I was not at all happy how this little teaser level experience had turned out. In fact, I remember thinking how unfortunate it was because I would NEVER attempt another teaser level again because the feedback from the community was almost entirely useless, and, if I took it literally, would give me very bad feedback about future games even if they were awesome.
Time has passed and wounds have healed. I knew then that I had failed to manage people’s expectations and I know this is even more true now. After I made the changes to the title screen, the score started creeping up and it has settled at 3.74. I really wish I could see statistics about how many 1/5s there are because I think it would be very telling.
The game has hit almost 1.2M plays after three weeks, which for a teaser level is sort of amazing for me. Despite the bad rating on Kongregate, I feel like it has achieved all of its primary goals. We got tons of valuable feedback and bug reports for very obscure edge case bugs. The iPhone port of Robo Riot got as much exposure as I could have hoped for. I don’t think I’m ready to give up on the idea of small teaser games or previews, but I do know that I’m going to approach them a lot more carefully in the future.



Spud
Regardless of how good your game is, kiddies on kongregate will dis your game just for the hell of it.
June 15th, 2009 at 1:20 amRegis
Gamers are junkies looking for a quick fix. Beware! People who are experienced in game testing provide MUCH more valuable input.
June 15th, 2009 at 5:29 amEnzo
After reading your post I’m left with the impression that game descriptions are mostly useful for SEO and not the gamers. I noticed on google webmaster tools that I ranked well for two games that are fairly old maybe its because I wrote good descriptions instead of taking them from mochi. BTW your game looks really good, have you thought of a downloadable as well?
June 15th, 2009 at 11:48 amurbansquall
Enzo: I’ve gotten burned with the downloadable market a few times so at this time, no I’m not really considering downloadable.
Regis: That is definitely true. That being said, being able to get a prototype in front of 200,000 people for some feedback is a reality in which all games are better as a result. I’d like to think it is possible with the right amount of expectation management.
Spud: True dat.
June 15th, 2009 at 1:59 pmickydime
I got to admit I played it and at the end I had a WTF moment… but then took 5 seconds and figured out it was just a teaser.
Its amazing how little people are willing to read (people includes me too, as seen from my WTF moment). We (the basement and Squize) have a game in beta right now and I sent it to some friends. You can throw or drop cards into their correct spots… Instructions page before you hit play clearly state this. But in the Q/A afterwords only half the people realized you could throw. disappointing… but learned that next time in game instructions might be better.
But in your case the glass is half full! Million hits for a teaser is incredible. And the people that complained about not enough levels obviously wanted more. With that attitude, they shouldn’t be turned away from trying the full version. I know I’ll be playing. Congrats and great work.
June 17th, 2009 at 8:04 amVlad
Flash gaming is clearly the fastest hit or miss game market there and gamers of free flash games want to put their hands on, play the game and move on. When of the very cool things you did with Nemesis was the tutorial and that explains it quite well: don’t make something for simple reading and you’ll do fine.
Problem with the sneak peak is that no one is going for the description reading, the same way they don’t want text tutorials. Your good will faced an expectation issue like you stated.
Still, you are doing an amazing work, so this is just an episode, keep your chin up!
June 22nd, 2009 at 6:20 amAP
I do not know how to use Flex Educational. I am lost. I do not know what that has to do with you. Amazing work, though. If you could help me. My e-mail: antor_paul@yahoo.com. I like helping people, but here I know nothing. I’m new to programming and only know about building web content so far, so fill me in please! I agree with Vlad. PS: Is your name based on Danny Phantom or is it real? If it’s real, I like it.
August 13th, 2009 at 9:22 pm