So like, what is up with play testers?

BlindSide · Post by **BlindSide** » Sat Oct 18, 2008 2:04 am

Ok I was reading this article on Gamasutra about the post-mortem of Age of Empires: Age of Kings, and here is an excerpt that shocked me about the play testers:

This item is a catchall for several problems that we encountered. On the good news front, our new offices have a dedicated play-test area equipped with identically configured machines. The bad news is that we didn’t make the best of it.

Many of our play-tests were not organized and focused enough, seriously reducing the amount of new and meaningful feedback obtained. It wasn’t always clear when we were testing for specific bugs and issues, and when we were testing for “fun.” We had a schedule of participants which drew upon the whole company, but schedule conflicts and lax enforcement resulted in the same people playing most of the games. We played too much multiplayer and not enough attention was given to the single-player game. And some people took it much too seriously, trash-talking other players, celebrating wins at the loser’s expense and storming off when they were losing. Play-test problems weren’t confined to Ensemble, though. At Microsoft, it was discovered that a play-tester had turned cheats on, playing to win not to test, in almost every game for over a month, which invalidated all the feedback from that group for the prior two months.

Aren't these people getting payed to do this? Atleast the Microsoft ones seem to do it as a full-time/part-time job, not just a side thing like the programmers/artists. Is this considered normal? I'm sure theres a lot more honest people out there willing to get payed to play test games, why are they stuck with those losers?

lostclimategames · Post by **lostclimategames** » Sat Oct 18, 2008 4:01 am

Well I think its the context that media, and 12 year olds put the job. It's glorified as being able to "play games all day" when really it should be considered is "tedious repetition that makes you hate videogames" which is why most dont even understand their job description.

torleif · Post by **torleif** » Sat Oct 18, 2008 5:59 am

Why where they bad mouthing the play testers? Debugging really should be up to the programmer. Also Age of Kings was the best AoE

If they didn't pay them it's even better. When bureaucrats over-think the design stage and listen to the beta testing 100%, you end up with the game that's unimaginative. Talk to any beta tester about what they want in a game, the'll say "Like counter strike but better" or "A MMO with really cool graphics". It undercuts all of the layers of the deign of a game, if a game where a cake the'd be looking at icing.

That's why so many production expensive games are flops, but simple games that are fun are success.

rogerborg · Post by **rogerborg** » Sat Oct 18, 2008 8:33 am

torleif wrote:Debugging really should be up to the programmer.

Winner. 100% code coverage via unit tests, 100% of the time. Write tests before writing the code that they'll test. If you can't be bothered doing that, then it's a bit rich to criticise playtesters for not doing their job either. That said, the evil usually starts with 'designers', many of whom couldn't get a job doing typesetting for kiddie comics, let alone writing comprehensive feature specifications.

Aaaaanyway, rants aside, the root cause is that nobody[1] actually wants to be a playtester. IME, playtesters are either skanky stoners who just want to play games, or people passing through on the way to more interesting jobs in design or development. They're the waiter-slash-model-slash-actors of the games development world.

[1] Rounding.

Dorth · Post by **Dorth** » Sat Oct 18, 2008 11:29 am

Wow. I'm disagreeing with Rogerborg. I'm. disagreeing. with. Rogerborg!

That's got to be a first.

Anyway, for having done the tester bit for a while and seen both from the inside of a big studio (EA) and a testing company that was really dedicated (Bug-Tracker, http://www.bug-tracker.ca/ ), I can guarantee you this has now evolve into a career in itself. You will get the gamers that wants to play and be paid. Those should be weeded out as fast as possible and it's HR's job to make sure they don't get in the place. You will get people (like me) that uses that job as a stepping stone to get into the industry (but also for other jobs, not just design, I went from there into a coder job, a guy at EA went sound designer, another level designer). This give you great insight into what might go wrong in every possible aspect of a game.

And then you've got the new breed. The guy (or girl, there are quite a lot) that is good at testing and loves doing that, doing it for 5, 10 or more years, perhaps his whole career. There are bugs an automated test will never find. Two textures in a randomly generated map next to one another that make an inappropriate image (or look like one). Game exploits that require balancing. And let's be frank, there will never be (or not in any forseeable future) proper and complete unit testing for EVERYTHING in a game. You have a game engine like Unreal that you pay for that has bugs. Which are regularly fixed, but still. And also, for which new features are regularly added, so it's not just bug fixes. Then you've got the in-house version, with in-house modifications, which will have some bugs. A short game development time (Don't ever expect a movie-licensed game to come out after a full maturation cycle. It's now or never). Human factor (stress, fatigue, memory lapse, firing/hiring, etc.)

You NEED professional testers nowadays. They shouldn't be telling you how to do your game. That's the job of a study group. Proper random polling of people inside the target audience at regular intervals in the development process. That's who you should be asking, THEY are the client (or talking for ""him""). Testers will tell you where bugs are, how difficult/easy a section is, will do the pre-certifications for a platform (you won't find a psp with an axis going in both left and right direction at the same time anywhere, yet it's a requirement that if somebody presses them both, nothing unusual happens.), localization (Can you positively guarantee that all strings that SHOULD be replaced in the game are? What about those that shouldn't? Is the translation proper?), compatibility (yes, that's part of QA's job description, making sure it works well on all targeted platform and variations. Trust me, it's tedious and you need a human eye to spot what's going wrong, because an inverted texture will not be registered in the most common automated test).

Seriously, some of the stigma touching the QA's job are plain disinformation or myths. Not all coders are 40 years old virgins living with their mom, covered in pimples, overweight, etc. It's the same for QA, they are not all gamers that want to be paid for playing. Nor are they all looking for a way to get in the industry as something else. Some are. But less and less.

Now, it seems to me the problem arose from a combination of the facts that some of those people were just ordinary joes with no testing background and one of their official (?) tester was an unprofessional twath. Problem is, how could he go on cheating for over a month without anyone catching on. Why wasn't the cheating logged? What was his lead doing? Shouldn't somebody outperforming the other QA by a lot be asked to give a demonstration or help others and thus be caught red-handed? Seriously, this really sound more like an HR WTF than a problem with using QAs...

BlindSide · Post by **BlindSide** » Sat Oct 18, 2008 1:41 pm

Two textures in a randomly generated map next to one another that make an inappropriate image (or look like one).

HAH! "Mommy! My Teuton formation looks like a penis!"

Why wasn't the cheating logged?

Yeah I'm curious about this too. It was a long time ago (1999-2000), so things have probably changed considerably in that arena. Is it also right to say that these days most newly released games don't seem to contain as much cheats/easter eggs as they used to? I mean AoK (Or was that AoE?) has that Mustang that fires rockets and stuff. I don't think many people these days are willing to put such effort into creating something that'll just make you say "Oh, that's neat.". But of course I'm being cynical and over generalising. (Its Generalising FireFox! I don't live in America!)

For interested parties the entire post-mortem is available here. It's a very insightful read into one of the most successful computer games out there. It's interesting how the company was just a small start up before AoE went gold, so they discuss some of the issues with sudden wealth/fame and dealing with the pressures of making something that has to live up to the first great success they made.

Cheers

rogerborg · Post by **rogerborg** » Sat Oct 18, 2008 5:10 pm

Dorth wrote:Wow. I'm disagreeing with Rogerborg. I'm. disagreeing. with. Rogerborg!

Not fundamentally, since I do agree that play/usage testers are absolutely necessary. And I'll bow to your recent knowledge of testing in the games business, since my experience is ten+ years old and from an indie house. I accept that big companies can attract and retain the filthy soulless pedants who comprise career button mashers. No offence, I love those guys and gals (women make the best testers because they know their limits), especially when I can close their bugs as INVALID (Read The Fine Spec).

There are bugs an automated test will never find. Two textures in a randomly generated map next to one another that make an inappropriate image (or look like one).

I wouldn't class that as a bug, and yes, that's exactly why you do need usage testers.

Dorth wrote:Game exploits that require balancing.

Painstick the designer that didn't prevent that in the design phase. It's their job to consider such interactions, and a massive waste of resources that they let such an exploitable imbalance through to testing. On the other hand, finding out how long it takes to level up is definitely a job for a play tester.

What matters is that play testers' time is spent on the right things: using the product as an end user would, and catching behaviour that genuinely couldn't have been anticipated, rather than catching bugs that should never have got past the whiteboard or compiler.

Dorth wrote:And let's be frank, there will never be (or not in any forseeable future) proper and complete unit testing for EVERYTHING in a game.

It's not inevitable. Unit testing is only skipped (in any environment) because programmers don't push hard enough to make it happen, or management doesn't support them when they do.

I didn't really believe in the merits of (unit) test led development until I experienced it first hand. It's a REDACTED revelation, and there's no reason why it shouldn't be used in games development, unless it's the explicit goal of the dev team to produce a series of buggy proof of concept demos rather than milestone builds of the final product.

When I'm king of the world, all development, everywhere, will be test led. Then we'll see. Oh yes.

Dorth · Post by **Dorth** » Sat Oct 18, 2008 7:16 pm

I DO agree 100% that test units should be conceived and used as often and as properly as possible. I LOVE test units. I think they're a godsend and if our industry is ever going to achieve the level of seriousness and dependability that other engineering sectors are known to have, we'll need to learn to use them and require that they are used. HOWEVER, like I said, I do not see this happening in a foreseeable future, because some titles by their nature will always be rushed and the industry as not yet invested enough into unifying base tools and concept. Why do we have a DirectX? Why can't our application be portable by nature or if they are (interpreted language), not have the same behavior on all similar platform (Java anyone)? We can't expect (nor want) a monopoly in the industry, it just hurts on the long run, but when you see bridge building engineer, they are more concerned about getting their formulas right and factoring in all that can happen than choosing whether to use this or that material. This is already decided the moment their calculations are over. One material IS better in one situation and that's the one that should be used. Until in our industry the same mindset sink in, the tools will always lag behind, causing rush jobs on a lot of project, leaving us with a lack of test tool and the need of professional testers. ^^

Don't worry, I'm still a Rogerborian

dlangdev · Post by **dlangdev** » Sat Oct 18, 2008 8:06 pm

You guys work at the QA department, obviously.

Maybe it's good to have shippable games defect free, just look at the market and name those companies who have listened to the mantra.

I'd rather shoot a fixed target with a date on it.

So yeah, QA testers normally gets crushed when the two humps meet. It's just the way it is.

So what do testers do? They simply do their best given the situation.

Dorth · Post by **Dorth** » Sat Oct 18, 2008 9:02 pm

Wha-hun?

wyrmmage · Post by **wyrmmage** » Sun Oct 19, 2008 2:22 am

So I'm curious: do you guys still believe in things like assert()'s, or do you believe everything should be done with unit tests? I know they're not meant for the same things, but there's a good amount of people out there that will only use assert()'s and not unit tests and vice-a-versa, and I've always been curious why

Dark_Kilauea · Post by **Dark_Kilauea** » Sun Oct 19, 2008 3:54 am

I'm fine with asserts in debug code, but I hate them in release. In release, the program should always try to handle every error gracefully.

Dorth · Post by **Dorth** » Sun Oct 19, 2008 5:32 am

While I agree with the base concept (when you can, DO recover gracefully), sometimes some errors just can't be recovered from and are too crippling. A blue screen in disguise with as much info as possible is often the best you can get then (a nice graphical window with text that make sense to a normal human and list the informations needed for that developers.)

rogerborg · Post by **rogerborg** » Sun Oct 19, 2008 9:49 am

(Roundup replies)

Changing the industry is going to require developers with the experience and confidence to insist on being given well defined requirements and writing unit tests for their own code. I'm occasionally tempted to dip back in, now that I have those attributes, but I fear that it would turn into a pissing match between my "do it right the first time" attitude and producers' "do something fast, then fix it, then change it, then fix it, then change it..."

asserts() are a fine tool, when used in addition to release build error handling. I use them:

0) As part of unit tests. I define a ReleaseAssert() for this purpose so that I can test release builds.

1) To help validate that control flow is deterministic.

2) As a debug build alert in addition to silently handling error conditions in a release build.

I strongly believe in assuming that anything and everything can and will fail, so all results should be checked and handled, even if it's just with (as Dorth says) a descriptive fatal error dialog. __FILE__, __DATE__, __TIME__ and __LINE__ FTW.

That said, I've taken a considered decision to not check the result of memory allocations on desktop games client applications, since an OOM indicates that the system is effectively hosed and needs instant user intervention. In those very narrow circumstances, I think it's better to crash out hard and fast to free up the game's memory so that the user can deal with it. Presumably the cause will be obvious, since the OS should be warning them about OOMs as well.

Embedded/console systems, server apps, and non game apps are a very different proposition, I hasten to add!

JP · Post by JP » Mon Oct 20, 2008 7:17 am

Man, play testing is such a chore! Loads of gamers think that working for QA would be an uber cool job, wow you get to play the latest games before anyone else! But you're actually playing mostly broken versions of the game and you're playing certain bits over and over to make sure they work and it just takes all the enjoyment out of it!

I did playtesting on LittleBigPlanet and Killzone 2 and though it was pretty cool to be playing them well before they were released and when it was only a few users playing them it got a bit of a chore when you had specific tasks to carry out and the bugs kept rearing their ugly heads meaning you'd have to start all over again....