Forums » General Pantheon Discussion

An issue with Pay to Test

    • 1019 posts
    February 5, 2020 10:07 AM PST

    I posted this at MMORPG.com....thought I'd post it here to...

     

    I believe an issue with current games and their "Support us at $1000 and become and Pre-Alpha tester" mentality is that you get Fan boi's that are doing the testing.

    Fan boi's aren't critical enough of the game, the design choices or the timelines for which it's been laid out.  A game being tested by people who won't question your choices is going to lead to a game that has bad mechanics, design implementation and more than likely game breaking bugs that weren't complained about enough that they are now forever in the game.

    The current MMO I play has patches regularly but some stuff breaks.  Completely breaks a game mechanic and when I, or whoever, voice our displeasure and want to know the actions and timelines, there are always people who say, "It's ok.  This or that.  Here's a work around.  Don't worry."  And if we were all to do that, say "It's ok", the game would suck.  It takes criticism to ensure a game is maintained or works properly and in the case of designing a game, it (any of them) should have critics playing the game from the earliest points of implementation.  But a critic sure as hell isn't going to pay $1000 to go in and criticize a game.

    Pay to test is great for indie game studios, but it's not good when the creators are only hearing praise for the work they are doing, slow as it might be.  Because a tester who is critical could end up losing their $1000 and not be an "invited" tester anymore.

    • 2419 posts
    February 5, 2020 10:14 AM PST

    Kittik said:

    I believe an issue with current games and their "Support us at $1000 and become and Pre-Alpha tester" mentality is that you get Fan boi's that are doing the testing.

    Fan boi's aren't critical enough of the game, the design choices or the timelines for which it's been laid out.  A game being tested by people who won't question your choices is going to lead to a game that has bad mechanics, design implementation and more than likely game breaking bugs that weren't complained about enough that they are now forever in the game.

    While I can certainly understand your position and accept that there are going to be such person who will not take testing very seriously, I can also say there is a sizeable percentage of people who have, are and will take testing very seriously.  Questioning everything, analyzing in great detail every mechanic; every interaction; every possible synergy...all in the efforts to ensure that exploits are caught early, that skills/spells/abilities are working as intended, that bugs are fixed, that content isn't broken, that design decisions are ones that are truly in the best interest of the long term health of the game.  I think we're speaking to the developers with far more authority, logic and reasoning than the fan bois.

    • 2130 posts
    February 5, 2020 10:16 AM PST

    Uh, no. I'm not really sure how much I'm allowed to say about this, so I'll keep it simple.

    This post is woefully misguided to say the very least.

    • 1019 posts
    February 5, 2020 10:24 AM PST

    Liav said:

    Uh, no. I'm not really sure how much I'm allowed to say about this, so I'll keep it simple.

    This post is woefully misguided to say the very least.

     

    Good to hear.

    • 1921 posts
    February 5, 2020 10:40 AM PST

    It must truly be the end times if I find myself agreeing with Kittik.
    What you've outlined has happened in every game I've ever seen, with or without payment involved, that has a limited audience for testing.  It's human nature.
    Critical expression is not encouraged.  It's a yes-men echo chamber of the highest order.  Negative feedback of any kind?  Ignored.
    Even if the financial investment wasn't an issue, the emotional investment definitely is.  Objectivity is the first casualty in these scenarios.
    No-one wants to hear they're wrong, especially developers.  And they're controlling the narrative here, so.. they're never wrong. :)

    • 2752 posts
    February 5, 2020 10:57 AM PST

    vjek said:

    It must truly be the end times if I find myself agreeing with Kittik.
    What you've outlined has happened in every game I've ever seen, with or without payment involved, that has a limited audience for testing.  It's human nature.
    Critical expression is not encouraged.  It's a yes-men echo chamber of the highest order.  Negative feedback of any kind?  Ignored.
    Even if the financial investment wasn't an issue, the emotional investment definitely is.  Objectivity is the first casualty in these scenarios.
    No-one wants to hear they're wrong, especially developers.  And they're controlling the narrative here, so.. they're never wrong. :)

    Yeah that isn't the case here. 

    Liav said:

    Uh, no. I'm not really sure how much I'm allowed to say about this, so I'll keep it simple.

    This post is woefully misguided to say the very least.

    QFT

    • 1428 posts
    February 5, 2020 11:03 AM PST

    life is a test.  i have to pay to play anyways.  i don't see a problem.  if one really wants and supports the development of the game, i think it's fine.

    there is a price to pay though.  question is, is the consumer in it for the long haul or a quick fix(seems more and more a trend nowadays thanks instant gratification)?

     

    this is all philosophical from a developer point of view of course:

    if i end up with a bunch of yes-man that want a specific type of product, sure feed the niche followers what they want.(maybe around a year for sustainability?)  btw this model makes a crap load of money with little risk other than destroying a company name(in which we could just bankrupt and start a new company with a different name), but hey i gave them what they wanted i'm not to be blamed.

    if i think purely just about what i would want to develop(assuming i'm a developer), i might not even secure funding for it or it becomes unsustainable and end up cleaning toilets forever.

    if i'm thinking about the longevity of the game(10+ years), there usually is a compromise that must occur between dev and players -which is pretty rare too since more and more people prefer non committal relationships.

    • 71 posts
    February 5, 2020 12:37 PM PST

    Kittik said:

    The current MMO I play has patches regularly but some stuff breaks.  Completely breaks a game mechanic and when I, or whoever, voice our displeasure and want to know the actions and timelines, there are always people who say, "It's ok.  This or that.  Here's a work around.  Don't worry."  And if we were all to do that, say "It's ok", the game would suck.  It takes criticism to ensure a game is maintained or works properly and in the case of designing a game, it (any of them) should have critics playing the game from the earliest points of implementation.  But a critic sure as hell isn't going to pay $1000 to go in and criticize a game.

    Pay to test is great for indie game studios, but it's not good when the creators are only hearing praise for the work they are doing, slow as it might be.  Because a tester who is critical could end up losing their $1000 and not be an "invited" tester anymore.

    When it comes to critics they tend to be just as bad as user reviews, only 1 out of a 1000 critics tend to be actually good at their job. I mean just take a look at the dispargy between critic and user scores on metacritic: https://www.metacritic.com/game/switch/pokemon-sword Yes, you see issues of differences in opinions all the time but when it comes to critics they're just paid to provide an opinion about a product and score it. Most of the time those scores are inflated due to either fan boi'ism, payment recieved for reviewing said product, or just the critic feeling they know more then everyone else (most of the time it's cash and the last one, the fan bias only happens with a small majority of I.Ps). So as i tell everyone offline: don't put a critic in high regard as they're no better then user reviews most of the time. 

    When it comes to user reviews though, yes it's prone to recieving influxes of both overly positive and overly negative praises from all walks of life. But within that you'll normally find reviews of the product that's even more informative then a critics take, example: https://www.amazon.com/gp/customer-reviews/R3TST2TH1PH7PI ; vs  https://www.ign.com/articles/2018/08/25/world-of-warcraft-battle-for-azeroth-review and in those reviews you can spot a vast difference in style and approach, one just gets to the point while the other rambles on and tries to drag out the review for as long as possible. But not only that, the critic sings more praises about the product while the user review sings more about it's faults. So, again proving that users can be more critical about a product then a paid critic. 

    Why is all that important? Simple really: 
    Those that have invested in this product (aka essentially everyone on these forums) want to see this product succeed but we also want the product to be enjoyable as well. The majority also don't want to wait 20 years like certain other mmo investments but we are willing to wait long enough for the game to progress nicely, which it is already mind you. Most that will play the next Pre-Alpha state will understand that the game is still in development and that feedback would be nessary in order for the improvement of the products state at launch, as we aren't "paying to test" but rather "investing in the product" and hoping our investments will prove worth while. If the end product ends up looking or playing worse then it did before launch then yes, our investement would be essentially for naught. But also if the product fails to meet exceptation or ends up shutting down after a year of release then our investments will also be for naught as well. I'm still abit sad about my time and money investments given to Wildstar, so I know all to well on how that losing such investments hits one. So most will be overly critical about the progression of the product due to the fear of our investments being for naught and you can find such harsh critisms on these forums alone. From users thinking they are being too silent or not showing enough progress (which they addressed in the last 2 streams) to users providing feedback on recent reveals of features like the spellbook to even old features such as class symbols. So again, there is no fear of users being "too nice" with critisms due to our own investments with the product. 


    This post was edited by znushu at February 5, 2020 12:39 PM PST
    • 844 posts
    February 5, 2020 12:57 PM PST

    Kittik said:
    I believe an issue with current games and their "Support us at $1000 and become and Pre-Alpha tester" mentality is that you get Fan boi's that are doing the testing.

    Of course they're fans. Nobody else would pay $1K to 'play' some early development game.
    Kittik said:
    Fan boi's aren't critical enough of the game, the design choices or the timelines for which it's been laid out. A game being tested by people who won't question your choices is going to lead to a game that has bad mechanics, design implementation and more than likely game breaking bugs that weren't complained about enough that they are now forever in the game.

    Disagree. I think fans can be the hardest critic's. And anyone paying 1000 large might feel even more entitled to *****. I know I would. Frankly you have no idea what PA people are saying, and that anyone even cares.

    Kittik said:
    The current MMO I play has patches regularly but some stuff breaks. Completely breaks a game mechanic and when I, or whoever, voice our displeasure and want to know the actions and timelines, there are always people who say, "It's ok. This or that. Here's a work around. Don't worry." And if we were all to do that, say "It's ok", the game would suck. It takes criticism to ensure a game is maintained or works properly and in the case of designing a game, it (any of them) should have critics playing the game from the earliest points of implementation. But a critic sure as hell isn't going to pay $1000 to go in and criticize a game.

    Again, you just role out with your big 'assumption' that 'pay to test players' will not complain and *****. Again you have no idea what the reality might be. And unless you've been involved with how games get designed and coded, you're just talking out your butt.

    Kittik said:
    Pay to test is great for indie game studios, but it's not good when the creators are only hearing praise for the work they are doing, slow as it might be. Because a tester who is critical could end up losing their $1000 and not be an "invited" tester anymore.

    I feel you have sufficiently overblown the entire idea that somehow psychophant PA 'pay to test people' are licking the dev's boots, and not being critical.
    I feel safe to assume that these PA testers biggest input is from filling up log files and doing needed load tests. Not their feedback, not their bitching. Just generate data.
    The fact that people are willing to pay for the log filling, load testing service is wonderful for a small studio.

    • 303 posts
    February 5, 2020 12:58 PM PST

    This forum may or may not be a good way to measure what the testers think but on here people want a one to one conversion of classic EQ. I hope VR have their own vision and I hope testers aren't afraid to express individuality in their feedback.

    • 844 posts
    February 5, 2020 1:15 PM PST

    Spluffen said:

    This forum may or may not be a good way to measure what the testers think but on here people want a one to one conversion of classic EQ. I hope VR have their own vision and I hope testers aren't afraid to express individuality in their feedback.

    EQ classic ain't being made again. It never will. If you didn't get to live it in it's prime then you never will.

    Even Brad didn't make EQ classic again when he started Vanguard. Good news is you can still get a taste of Vanguard on the emulator.

    • 1273 posts
    February 5, 2020 1:20 PM PST

    I'm not really sure why you have that opinion.  I have almost the oppositie opinion.  If I were to pay $1000 to back a game I'd have even MORE motivation to make sure it's developed correctly.  If I were to spend say...$40 to get early access to a game then I'd get the point.  But, at least for me, the more money I put into something the more serious I am about that thing.

     

    With that said...I'm actually fairly against the whole idea of "pay to test" or "pay for early access" too.  But for completely different reasons.


    This post was edited by Ranarius at February 5, 2020 1:22 PM PST
    • 303 posts
    February 5, 2020 4:57 PM PST

    zewtastic said:

    Spluffen said:

    This forum may or may not be a good way to measure what the testers think but on here people want a one to one conversion of classic EQ. I hope VR have their own vision and I hope testers aren't afraid to express individuality in their feedback.

    EQ classic ain't being made again. It never will. If you didn't get to live it in it's prime then you never will.

    Even Brad didn't make EQ classic again when he started Vanguard. Good news is you can still get a taste of Vanguard on the emulator.

    Hence why I'm worried about this echo chamber.

    • 287 posts
    February 5, 2020 5:10 PM PST

    $1,000 is quite a paywall. You would have to be very enthusiastic to get involved, and would want a game that is specifically molded toward that person's liking. I think a more practical concern would be that this "fan boi" wouldn't necessarily be a knowledgable tester that might not be able to provide detailed bug reports to help iron out bugs during testing. Even then though, the $1k itself helps production.

    • 159 posts
    February 5, 2020 6:30 PM PST

    Kittik said:

    I posted this at MMORPG.com....thought I'd post it here to...

     

    I believe an issue with current games and their "Support us at $1000 and become and Pre-Alpha tester" mentality is that you get Fan boi's that are doing the testing.

    Fan boi's aren't critical enough of the game, the design choices or the timelines for which it's been laid out.  A game being tested by people who won't question your choices is going to lead to a game that has bad mechanics, design implementation and more than likely game breaking bugs that weren't complained about enough that they are now forever in the game.

    The current MMO I play has patches regularly but some stuff breaks.  Completely breaks a game mechanic and when I, or whoever, voice our displeasure and want to know the actions and timelines, there are always people who say, "It's ok.  This or that.  Here's a work around.  Don't worry."  And if we were all to do that, say "It's ok", the game would suck.  It takes criticism to ensure a game is maintained or works properly and in the case of designing a game, it (any of them) should have critics playing the game from the earliest points of implementation.  But a critic sure as hell isn't going to pay $1000 to go in and criticize a game.

    Pay to test is great for indie game studios, but it's not good when the creators are only hearing praise for the work they are doing, slow as it might be.  Because a tester who is critical could end up losing their $1000 and not be an "invited" tester anymore.

     

    agree with most of what you said. I want people to be honest, the problem with this is. When you do give honest criticism of the game in hope of helping improve the game it always opens the floodgates for all the negative people that never have a nice thing to say. So the honest criticism IMHO gets losts as soon as all the drama starts from the haters and you get category / labeled as being a hater too. I try to give criticism when I feel it is needed and praise when I feel like they ( VR ) earned it.

     

    • 65 posts
    February 5, 2020 8:02 PM PST

    I've pledged several times(still under the pre alpha limit) because I want this project to succeed. I want Brad's vision to live on. But mostly I want to play a game that I will enjoy for years with other like minded people. I'm planning on being brutally honest when alpha testing comes around. Constructive criticism and praise, not brainless appeasement, will make Pantheon what we want. Why would I invest hundreds or dollars, time and energy into a game that I want to fail? 

    • 844 posts
    February 5, 2020 9:46 PM PST

    Spluffen said:

    zewtastic said:

    Spluffen said:

    This forum may or may not be a good way to measure what the testers think but on here people want a one to one conversion of classic EQ. I hope VR have their own vision and I hope testers aren't afraid to express individuality in their feedback.

    EQ classic ain't being made again. It never will. If you didn't get to live it in it's prime then you never will.

    Even Brad didn't make EQ classic again when he started Vanguard. Good news is you can still get a taste of Vanguard on the emulator.

    Hence why I'm worried about this echo chamber.

    How can anything like this not be an echo chamber.

    Everyone here is an enthusist in one manner or another.

    Most on these forums and in PA have paid to express their enthusiasm which is more than most game forums can say.

    With Brad gone, we are left with a studio of people, (none?) of which worked on EQ and probably not Vanguard either. Most probably never even played either as well.

    I'm not sure what the worry is all about. Making it too much like EQ, or not alike enough?

    Persistent world with no instancing is about as close as I need it to EQ and Vanguard to enjoy. All the other stuff can be a surprise. Just don't take 10 years.

    • 2756 posts
    February 6, 2020 1:14 AM PST

    As others have said, I'm not certain how much is allowed to be said, so I'll just say this...

    It may be a trend, but not all companies treat backers' 'testing' phases as a glorified early access. Those getting involved in 'real' pre-alpha testing will tend to be genuinely interested in helping shape the game and are not backward about coming forward where they see problems and have feedback.

    VR have said many times and shown in many ways they are interested in all feedback, positive or negative, as long as it is detailed, reasoned and constructive. There is no reason to believe this is different in testing sessions and forums.

    Me, I've always looked at these forums and thought how difficult it must be for VR to navigate between the hardcore EQ/VG purists and those more accepting of some modern MMORPG developments. Even though we are all here for a return to group-centric, challenging, social gameplay, almost every aspect of the game has diametrically opposing opinions.

    I don't doubt it's a problem for other games, but I don't get why anyone would think it will be a problem here. Just looking around in this forum you see many people being very critical of VR's choices or at least putting forward alternatives and additions. A lot of those same people are VIPs and pre-alpha testers.

    • 145 posts
    February 6, 2020 5:21 AM PST
    I understand your view point and it is some what correct criticism sometimes leads to improvement. But look at wow they tried to please all the critics initially and now won’t listen to any. Vr has to be critical of itself and yet stay true to its core tenets. As far as the 1000 pledgers I would say that they are most likely the most active people on these forums as they now have a vested interest in seeing that the game succeeds and becomes what they wanted it to be.
    • 520 posts
    February 6, 2020 7:59 AM PST

    I understand where you are coming from I do agree with Vandraad and zewtastic. If someone is investing a lot of cash he wants the game to be great and greatness was never achieved by praise alone - when there is time to give pat on the back fans will definitely do so, but I doubt that most of them would save the constructive criticism for  themselves. There will always be people that operate in zero-one system where only 0-1/10 and 10/10 exists, but they'd be there either way. Ultimately the decision of development fall on VRs laps and we can only hope that they'll make use of their brains and wont blindly follow what  pledgers want, but will consider "all"  suggestions and remarks and decide wether it fits their vision and will be good for the game in the long run. What gives me hope is that a majority of Pantheons fan-base are veterans that saw a lot and know what usually works and what doesn't and not youngsters like in most games. Being able to play during pre-alpha is a great incentive to support the game for people that are able to afford it, that would not pay that much otherwise - which is great for game development.

    • 1019 posts
    February 6, 2020 8:25 AM PST

    I'm often mistaken.  It's how I live life.  Yes, it's a crazy life I live.

     

    Anyway, I'm not saying those that Pay da monies aren't critical and that Fan boi's aren't critical. 

    But I feel they are critical in a different manner.  And often times are more of a yes man and a cheer leader than a clear eye skeptic of the game.

     

    Take the sailor who spent $25k on a boat that never works, he curses at that boat and is always trying to fix it.  He keeps putting money in that boat as it is sinking and his mentality is I can make it work...he keep putting money toward the boat, keeps working on it and ignores everyone's pleas to leave the boat.  But he stays until he drowns. 

    Where as a person who has put nothing toward that boat is yelling at that guy to stop putting money on that boat and then throws the guy a life vest.  

    Super weird analogy and I'm not saying Pantheon is a sinking boat.  What I'm trying to say is if you put money in something you immedatly look at it differently.  You look at it like you can do something, that it's worth just a little bit more.  

    I've invested in Pantheon myself and this post isn't directed at VR or the devs....this is just the forums I mess around on the most, so I thought I'd take a mind dump here.

    • 3852 posts
    February 6, 2020 8:37 AM PST

    ((Fan boi's aren't critical enough of the game, the design choices or the timelines for which it's been laid out. ))

    This is half right. We are *not* typical of the larger community. We know a lot more about MMOs in general and Pantheon in particular before even getting into the first day of a test. Things that may baffle a true newcomer may get unmentioned because to us this is so normal we don't even notice it. Things that might make a true newcomer to the game blanch like having mobs stolen if the system is most-damage-done or being killed by other players' trains on a pve server may not get our comments because "that is how EQ or Vanguard did it so of course it is how it should be done" or "oh yeah that was discussed 50 times on the forums why even mention it?"

    On the other hand we are attuned to many issues, some trivial, and have strong opinions. We will comment at length on things a newcomer wouldn't notice and wouldn't care about. We want the game we are emotionally invested it - in some cases for quite a long time - to be perfect not just good and will comment accordingly - more strongly than an impartial tester would. So you are also half wrong.


    This post was edited by dorotea at February 6, 2020 8:38 AM PST
    • 1281 posts
    February 6, 2020 1:56 PM PST

    Kittik said:

    I posted this at MMORPG.com....thought I'd post it here to...

     

    I believe an issue with current games and their "Support us at $1000 and become and Pre-Alpha tester" mentality is that you get Fan boi's that are doing the testing.

    Fan boi's aren't critical enough of the game, the design choices or the timelines for which it's been laid out.  A game being tested by people who won't question your choices is going to lead to a game that has bad mechanics, design implementation and more than likely game breaking bugs that weren't complained about enough that they are now forever in the game.

    The current MMO I play has patches regularly but some stuff breaks.  Completely breaks a game mechanic and when I, or whoever, voice our displeasure and want to know the actions and timelines, there are always people who say, "It's ok.  This or that.  Here's a work around.  Don't worry."  And if we were all to do that, say "It's ok", the game would suck.  It takes criticism to ensure a game is maintained or works properly and in the case of designing a game, it (any of them) should have critics playing the game from the earliest points of implementation.  But a critic sure as hell isn't going to pay $1000 to go in and criticize a game.

    Pay to test is great for indie game studios, but it's not good when the creators are only hearing praise for the work they are doing, slow as it might be.  Because a tester who is critical could end up losing their $1000 and not be an "invited" tester anymore.

    I disagree.  I can't vouch for anyone else, but *I* believe that true "fans" will be MORE critical of the game during testing rather than less critical.  They will be less likely to let things slide.  I paid $500 for my Pledge, and you can bet your ass I am going to be critical of the game.

    • 1303 posts
    February 6, 2020 3:04 PM PST
    For my own part, I didn't pledge what I did in order to test or to get in and play early. I pledged to assist in funding a project that I hope will eventually become a game I actually want to play post release for years to come.

    That being said, I do intend to take testing seriously. I don't really see a point in participating during testing if I'm not invested in helping to make things better. And the more I pledged the more true that position becomes.
    • 1785 posts
    February 6, 2020 3:06 PM PST

    I can only speak for myself, but here is my opinion on this topic.

     

    As a supporter, I chose to give the amount of money that I did to VR because I agreed with the overall vision that was outlined on the website and being discussed among this community on these forums, I wanted to see the game succeed, and I had enough discretionary income at the time to support a pledge of that size.  Being able to get into a testing phase really did not play into my decision at all.  Even by the time I did pledge, it was obvious that Pantheon was going to do many things that I agree with and support and likely some things that I do not, as well as a great many things that are somewhere in between those two extremes.  I accept that.  I'm not the one making the game after all.

    For my part, my goal in any feedback that I provide to the developers is for Pantheon to be the best experience possible for its players.  This is true whether that feedback is in a testing session, here on these forums, on twitter, or in any of the various community sites and discords that I participate in.  Obviously, my feedback will always be biased by my own experiences and preferences, but that's ok - because everyone else is providing feedback as well, and their perspectives will be slightly different from my own.

    I do not believe that my pledge level makes me any more or less objective or critical as a tester.  Only prior experience with MMORPGs and life can do that - and sometimes, that past experience may be as much of a hindrance to seeing things objectively as it is a help.  All of us, regardless of how much we pledge to Pantheon, are going to look at what's there and evaluate it against our own standards.  If there is one thing I can say about this community it's that we are passionate and vocal - and I sincerely doubt that any of us are going to remain silent if we see something that doesn't sit well with us, whether that happens in testing or elsewhere.  We may approach that differently depending on how good we are at written communication, and how tactful and diplomatic we feel like being, but if there's one thing that all of us are good at it's voicing our opinion.

    The fact that sometimes people in our community disagree isn't proof that anyone is being a "yes man" or a "cheerleader".  It's simply evidence that we have different perspectives and different things that are important to us.  When that happens, it's VR's job to look at ALL of the feedback and try to find the correct path forward - which, honestly, is usually somewhere in between the different viewpoints that we're all expressing.

    Our feedback is very important to VR now, just as it will be during Alpha and Beta.  Prior to launch, VR can still make changes based on our feedback at any point.  Even after launch they can make changes, even if those changes may be a little slower to arrive.  How much we pledged or when we get to provide that feedback is FAR less important than the fact that we do.  The fastest way for Pantheon to be a terrible experience is if we all stop advocating for change when we think it's needed.  We need to accept that we won't always get things exactly the way we want them, and we also need to accept that other people's feedback and opinions are just as relevant and valid as our own.  We need to accept that sometimes no one is right and everyone is wrong, and that's ok.  The important thing is that we all need to respect each other, instead of labeling people that we don't agree with, or trying to say their opinions aren't valid because they pledged more or less than we did.  Not only are those things childish, but they actively work against Pantheon's success by giving people the wrong impression of our community