Public vs. Closed/Private Beta Testing

[Lutz]

Which do people prefer, and why?

Personally, I have never released an alpha/beta/etc version into the wild for testing -- it seems to me that it might result in so many wildly disparate opinions as to make the feedback potentially unusable (plus, if I'm honest, I don't want to release something that isn't as good as it could be). Still, lots of folks on this forum do just that, so there must be some benefits; anyone willing to enlighten me?

On a related topic, Doom II - Hellscape is about ready for testing.

[Memfis]

Private is cooler imo. It feels more professional or something. I don't like when all the dirty laundry is here for everyone to see. That gives the project a less serious sort of feel. I prefer when projects have a bit of mystery to them.

Also many people will play your beta, and then when you release the final version they won't replay it. So many people will remember the inferior version of your wad and will never even see it in its final form. I think that can be sad.

[Voros]

First private. Get a small group and test it among themselves.

When you believe it is ready for a public release, then put out a public test.

[scifista42]

Lutz said:

Still, lots of folks on this forum do just that, so there must be some benefits; anyone willing to enlighten me?

1. It gives a chance of getting a lot of feedback from the most wide range of many different people, which is usually what authors want (to improve their work, to see the reactions, or for other reasons), even knowing that it's just a chance and that reality might let them disappointed about both the amount and the kind of feedback they get.
2. It doesn't involve contacting (that is, bothering) any particular person with a request for help, but leaving the task to those who spontaneously decide to try the wad out based on how it's publicly presented (that is, it's all comfortably casual).
3. If the wad was announced before and it already has fans who eagerly await it (or if the released wad's presentation is so awesome that it gains such fans instantly), releasing even early content usually make many of them happy, some of them so much that they will even eagerly provide feedback as to allow the author to improve the final work, and then the author can be happy both for making them happy and for the feedback they give him.

I always go from testing by myself-only straight to a public (beta) release, but I'm also no great mapper and have no high ambitions in that field, which possibly excuses me from what may be called "responsibility" for the quality of the final work. But I certainly like the concept of public testing more than private testing, both as a player and as a mapper.

[Vorpal]

It's a question of numbers I think, not public vs. closed. It's true that yes, more eyes will result in the eradication of more bugs. However it's also inevitable that you'll get more people lobbying for changes to X/Y/Z, and it'll never stop because each tester has their own X/Y/Z list. So it's like you make a stylistic/visual/flow/whatever alteration to satisfy one tester, then another tester hates the change and lobbies for a change to the change. It has the potential to delay for weeks/months and can suck the character/individuality out of a design as you try to please a huge audience with a thousand compromises.

I'm rather fond of leaving the task to a handful of friends that I trust, and if there are any warts that remain, I will just chalk it up to the "character" of the map (with exceptions ofc)

[SaladBadger]

I do a bit of both. I will often distribute partially completed copies of my maps to friends to ensure the gameplay isn't complete garbage and the map flows okay, but once I have a final map that's been signed off on by private testers, I'll generally package it up for a formal release and dump it here. While these releases are usually "complete", I'll take any notable feedback I get and do bugfixes as needed. Once all of this is done, I'll put the finalized version on idgames.

I've done this with Moonsong and Moon's Call so far and its worked out fairly nicely.

[lil'devil]

I think you should do both of them: private beta testing first and public beta testing later.
Atleast that's how many developers in video game industry usually do. At first, they get a selected few people for closed beta testing and then, some time later, they give away a bunch of keys for an open beta testing, where everyone can participate.

[Keyboard_Doomer]

Open development is much better for avoiding bugs and shaping the final product in a way that it won't have gameplay or design choices that may turn out to be questionable or downright unenjoyable. This approach is more beneficial the sooner you make the development open - you'll get the best results if you release the maps one by one immediately after you finish them. If you just release 32 maps at once in a beta you're losing most of advantages of this approach.

It can also motivate you to continue the work when you're nowhere near the finishing. If you're barely half-way and you see just how much hard work did the first half take it can be really daunting especially if there's no one to appreciate the hard work. But if you have people from the community who help you with each step it's likely to feel much easier.

Of course, open development has some big downsides which you and Memfis already outlined. If someone already played an earlier version of the map, they're quite unlikely to play it anytime soon again in its final form. In the feedback for the maps in development you can get easily flooded by subjective critiques and may feel pressure to change something that might not even require a change from an objective point of view. Also, it seems to be widely considered "cool" if you can just come up with a final product which required hundreds of hours of work rather than openly showing that you're a human that needs some help and encouragement throughout the development process.

The way to avoid the woes of open development while maintaing similar advantages is to get from the beginning of the project a group of competent and dedicated playtesters using a range of different source ports. That can, obviously, be pretty difficult in the relatively small Doom community.

[Gez]

Public testing, I may download and check and then report my feedback. Private testing, I'd have to ask, and then I'd feel compelled to test and report, and since I'm not sure I'll have the time to actually do that then I just don't ask.

[Not Jabba]

Lutz said:

Which do people prefer, and why?

Personally, I have never released an alpha/beta/etc version into the wild for testing -- it seems to me that it might result in so many wildly disparate opinions as to make the feedback potentially unusable

I'm generally of the opposite opinion re: feedback. Too few testers means you can't see any trends in opinion -- you just get like three different people telling you to do three completely different things. If you get feedback from 20 people, you know to be concerned about the issues that were raised by many people, as opposed to just one or two who happen to be outliers. If you're going to use just a few testers, they have to be *really* good testers who can see what you're trying to do as opposed to what they wish you'd done -- not that you ought to have any trouble finding people that good.

It also definitely depends on experience level. A relative beginner should post a public beta for general feedback, because they probably have a ton of stuff they need to work on. For someone of your caliber, it seems like the most logical thing might be to get a few experienced people together who roughly share your design philosophy, communicate to them what you're going for, and see what they think about whether your levels actually meet your vision (and catch any obvious bugs). Then when you're happy with what you have, like Voros said, open it up to the public for more thorough bug stomping and to see if there's any gameplay that's widely loathed that you might want to tweak.

[Memfis]

Vorpal said:

and can suck the character/individuality out of a design as you try to please a huge audience with a thousand compromises.

This is a very serious issue in my eyes. I remember how valkiriforce was overwhelmed by the testers' feedback and he said that he felt like Reverie was becoming more and more different from what he had in mind. That sounds pretty scary to me as a mapper, I would hate to experience something similar.

Like yeah, sure, you don't have to listen to everyone. And you can only perform changes when people bring up valid points that you 100% agree with. But even that is not necessarily a good thing I think. I often find myself being more attracted by somewhat raw and unpolished works that have some issues, which I don't necessarily see as problems but more like unique traits that give these works more character.

Then there are such problems as certain testers participating in so many projects that you can feel their presence virtually everywhere. That also can be dangerous for the community as a whole I believe. Maybe it would be nice to have more popular testers with very different tastes so that they could kind of balance themselves out. Or maybe that's already happening, I'm not really that familiar with the testing scene (is that even a thing?).

Finally, there are weirdos like me, who liked some early alpha version more than the final product and they will bitch at you for ruining a cool project with changes that are completely against the original spirit or whatever.

[baja blast rd.]

Memfis said:

This is a very serious issue in my eyes. I remember how valkiriforce was overwhelmed by the testers' feedback and he said that he felt like Reverie was becoming more and more different from what he had in mind. That sounds pretty scary to me as a mapper, I would hate to experience something similar.

As a mapper, I want my works to be objectively strong, and I view feedback primarily as a means of learning about problems that I overlooked or didn't consider. Sometimes these are bugs. Sometimes they are weak links in fights -- e.g. a simple strategy or approach that makes things much easier than I wanted them to be. In a recent map a cacodemon apparently landed on a key, and the player got lost for a while. I never wanted that to happen in the first place.

What I'm not doing is making changes that are against the spirit of my intentions. If I make a section with no health and someone says they got there with 15% HP and had to reload countless times, well, since I don't really care to calibrate my map to be accommodating of people who save with low HP, I'm not going to take the sting out of the section by adding medkits, no matter how many people suggest it, sorry.

I think it's important for mappers to be strong-willed in such cases, to establish limits. So valkiriforce's case is unfortunate, but that isn't a problem with the advice itself. At some point they have to put their foot down and say no.

Personally I never take offense when my suggestions aren't adopted. It doesn't piss me off; and I don't 'accept it', nodding to myself morosely and saying 'okay, that's fine, the mapper doesn't have to do everything I say'. Rather, it doesn't really register at all as a 'thing'. I hope my suggestions are thrown out when they don't fit the mapper's vision at all. I would feel very guilty about giving feedback if it turned out that a mapper did what I said because they felt obligated to, or because they felt 'the tester always knows' or some sort of nonsense like that.

But I think I see where you are coming from. For example, the Post Your Pictures thread in Doom General is completely obnoxious to me. It's not just because it seems like a few people take themselves too seriously there :D. It's mostly the sense that mappers are really making changes to their works based on the often excessively nitpicky, "too many cooks"-y, or sometimes plain bad advice.

Memfis said:

... that you can feel their presence virtually everywhere.

How so?

[esselfortium]

I like private testing. Knowing your testers and having faith in their recommendations (and being comfortable debating with them!) is conducive to getting work done and decisions made, whereas having countless outside voices telling you inevitably-contradictory things just makes for a lot of stress and uncertainty IMO.

Have a good idea of what you want your map to be, know your testers and their preferences, and consider their advice where it lines up with your goals. It's nice to have a small group of experienced testers who can coherently explain what they consider issues and propose alternative ideas in detail.

If you're making a map that's far outside of your usual comfort zone, it's a good idea to seek advice from folks who are more experienced with those niches, to confirm that you haven't made any glaring oversights.

On long-term projects, it can be helpful to bring in new testers now and then to get a fresh perspective, since it's easy to overlook things when you've been testing and revising the same maps for months or years on end.

Because it's hard to call anything truly done, I inevitably still call it a beta when releasing things half the time, and then keep working on it more...

[Deadwing]

I prefer public in my case because I want to improve myself on my mapping skills, so getting a fairly good amount of feedback helps a lot balancing stuff or being sure of more "experimental" decisions. Private testing works better if you're more experienced and have people avaliable to do the tests.

[40oz]

I think project transparency is better than private testing. I'm sure there are members in this community who can be excellent testers but don't post often or network with people enough to be known for it.

I've been working on a "how to be a good playtester" tutorial that is still in development but hopefully it will circulate a little bit enough to make the general public in the community produce more reliable feedback that mappers can use. It's unfortunate that there are junk opinions in the community and that mappers have to be selective of who they want to listen to.

Congrats on Hellscape btw, it looks like its gonna be awesome

[Lutz]

Wow -- great responses, here, folks; thanks for taking this question seriously.

[scifista42]

rdwpa said:

the Post Your Pictures thread in Doom General is completely obnoxious to me. It's not just because it seems like a few people take themselves too seriously there :D. It's mostly the sense that mappers are really making changes to their works based on the often excessively nitpicky, "too many cooks"-y, or sometimes plain bad advice.

Blame those people, not the thread as a whole. It's awesome to see pictures of works in progress.

[dobu gabu maru]

Congrats on getting Hellscape ready!

This is a good question I had originally wanted to make a post about myself. I think there's merits to both sides, especially depending on what kind of project you have in the works. If it's something more personal or unconventional it's probably best to find testers that fit its style, whereas something you want everyone to enjoy is best released to the public IMO. Of course, you also have to factor in what kind of an impact you're looking to make with its release--some projects that have been accumulating maps over time in an active thread here are more suited to public participation, whereas if you've been working on something in secret it can carry a lot of weight if you go the private route and get it "archive ready" when you're finally ready to show it off. You're more likely to get a greater amount of attention if its a product people feel they can download, play, and won't have to delete somewhere down the line, but you also run the risk of making that project too insular, where perhaps a lower difficulty is too difficult or the playtesters didn't catch a bug in some early version of a specific port.

I'm fine with either personally; I like checking out things before they're finished but don't mind waiting for the 1.0.

[Capellan]

Voros said:

First private. Get a small group and test it among themselves.

When you believe it is ready for a public release, then put out a public test.

This, generally. I prefer not to throw things out to publicly until they've had at least some playtesting and polish, but the wider audience will often find things that even the most dedicated small team can't.

[MrGlide]

Very valid points have been made for both sides, but one point I would like to add is private testing only works if someone is comfortable enough with voicing their opinion to the author in a respectful and constructive manner. If private testers can't give constructive criticism because they're in fear of hurting feelings, then essentially all they're doing is bug testing.

[gaspe]

Public testing is fine imo but only if you release the whole mapset, or at least a good chunk of it. I probably learned it too late with my project but it seems that releasing progress as you make them is a rather unsuccessful business :P in particular I feel a bit like this:

Memfis said:

Also many people will play your beta, and then when you release the final version they won't replay it. So many people will remember the inferior version of your wad and will never even see it in its final form. I think that can be sad.

[Albertoni]

My take on this is: Each mapper has a certain style and usually a certain thing they want to achieve with the map. Working in public might get enough criticism that you'll change things to fit other people's visions instead of yours, thus ruining what you wanted to do.

So, I test it with few, close friends who understand what I'm trying to do until it's ready for a real-world test. Then I proceed to partially ignore all advice that doesn't fit with what I try to do. I still listen to them, but if my goal was to make an empty wasteland which you must explore with only half a dozen monsters, I'm not adding more, no matter if the entire thread is begging for it.

[glenzinho]

If this is about your new cool mapset and I'm not going to get an advanced copy, then i'd definitely recommend public testing :^)

[dew]

Both have advantages and disadvantages, community projects will obviously favour openness and networking while team projects favour in-house testing, because they want to roll out a complete product. A somewhat balanced approach can be stress-test your maps in early/alpha stages with a few testers privately, then do open beta and RC testing. This way you can make big changes in structure, experiment with gameplay ideas that may or may not pan out well... and run it against a small test sample instead of hearing the same whines for a month from thirty billion people. Then when you have your map "stabilized", you can let the public have at it. Dealing with "scrap the entire north section and start over" in the public is a sort of self-inflicted, easily avoidable harm, heh.

[Steve D]

I like a mix of public and private. I'm fortunate to have 5 or so people I can lean on for testing, whose opinions I respect. Usually, I'll try them first for private testing. Once they've gone through the maps and I punch things up, I'll open a thread for public beta.

Sometimes, I'll go public first. For example my Shotgun Symphony episode, which went to public beta with little or no private testing. I got some great advice from my regular testers like Demon of the Well, who was joined by some folks who hadn't playtested for me before, such as Dobu, Magnusblitz and scifista42, all of whom provided valuable commentary. I also ran into plums, who did great testing and also instructed me on titling and other practical issues I was ignorant of.

I like to point out that scifista42 read me the riot act on that mapset about its lack of polish. I think maybe he believes I'm messing with him when I bring this up, because after all, like I told him back in 2014, 6 of those maps were from the '90s, and I was going for that '90s feel, an homage to Fava Beans except with much bigger and much more violent maps. But he pointed out that the polish level was inconsistent, and I eventually agreed with him. As a result, I polished the maps quite a bit, and I'm far more satisfied with the result. Thanks, sci!

I also got really lucky when John Suitepee livestreamed the mapset. He pointed out when one of the maps felt sloggy, in a big hallway that had some Barons and Cacos in it. This was towards the end of a map with about 400 monsters. So what I did was, I removed all the monsters from that area, most of the monsters in the next area, and concentrated on a bigger final fight. The end result was a significant improvement in the map's gaming flow.

This goes to the point raised by Memfis about mappers sacrificing their vision. I guess it all depends on what the vision is. My primary vision is just to make the best maps I can. How I get there is to some extent open to debate. If I feel the advice I'm getting will make a better, more fun map, I take it. One example would be the map I submitted as E4M2 -- Perfect Hatred -- for Ultimate Doom in Name Only. It was actually donated from a different project that died, and Magnusblitz said it didn't seem to have quite enough hatred. That's a very general comment, of course, but my reaction was, "You want some hatred? I can give you some hatred. How about a little awkward platforming over damaging floors, and then this dickish 3-Cyberdemon trap? Is that enough hatred for you? Or do you want more, because I can give you more hatred if that's what you really want!" ;D By the same token, scifista's comment about polish was very general. He didn't suggest any specific thing I should do to polish the maps, so all those decisions were mine, but he suggested that something needed to be done and he knew I could do it. So I did.

There's also been times when I took some advice and in the end decided it didn't turn out so well, so I changed things back to the way they were. I've also never changed anything that I felt was a core quality of a map, it's reason for existence, as it were. But the bottom line is that I've gotten great feedback from some cool people and they've helped improve my maps. Maybe it helps that while I take mapping seriously, I don't take it too seriously. As a result, I'm not easily offended by anything people say about my maps.

It's also true that I, along with most mappers, don't draw as many beta-testers as skillsaw does. ;D As a result, I don't have to sort through some bombardment of opinions from like 50 different players. I've never even had 10. So I think that worry is a non-issue for most mappers.

Anyway, I've seen the screenshots of Hellscape. I'm drooling! :)

[Jon]

I'm leaning towards semi-public because I think it might help with motivation to get things finished.