-
Ever wanted an RSS feed of all your favorite gaming news sites? Go check out our new Gaming Headlines feed! Read more about it here.
Indie Dev Rami Ismail explains the console certification process
- Thread starter Captain of Outer Space
- Start date
People here and on the internet really needs to learn about how looked down most of the time QA is, and how many of the things that made to the retail builds are already reported, but QA has little to no-power to stop a game-breaking bug to be closed if production deem it necessary to hit the deadline.
Q&A sounds like some really hard shit.
Like, you have to find problems and then tell a person that there is a problem in their product.
(I am not being sarcastic btw)
Like, you have to find problems and then tell a person that there is a problem in their product.
(I am not being sarcastic btw)
I've seen enough developers on here explaining before that there's always triage list of bugs, from the game-breaking ones to the ones with minimal gameplay impact, that all gets laid out in tiers. QA might know about it, but depending on the urgency of the release, there will be stuff that will be deemed sufficiently shippable and won't be fixed for launch, outside of the critical game-breaking stuff.
I've never worked on console games, but in the inter-personal aspect, it's not always that bad.
However, the more stress and the closer a deadline is, tempers starts to flare and it can get ugly at times. I'd worked with several devs and most are pretty cool about it, and understand that we finding errors in the game it's the nature of the job. But just as that, others can be real assholes when you point out bugs.
Honestly it isn't fair to blame producers either as they are often put in an impossible spot where they can't do the work but are being told to make sure something is finished by X date as they raise concerns about hitting X date. Problems like what is seen at CDPR start at the top. If their studio head was a good leader most of this stuff would be ironed out from there.
So fucking hard especially when engineers literally think they can do know wrong. Also important to note that most QA is made up of a bunch of hourly contractors that can be let go at the drop of a hat while the engineers are all full time employees that can't be fired very easily and aren't considered "easily replaced" like QA is.
For sure. I should say, I don't want to make it sound like every QA job is the pits. I've had a couple (including my current one) that I genuinely enjoyed, and worked with devs that I consider friends to this day.
I guarantee that same thing was told to them by a co-worker they hated. Then they saw it again from QA and they needed someone they could kick without reciprocation.
Anyone who thinks MS and Sony aren't responsible aren't realistic.
Even in the investor meeting cdprojekt was asked how were they allowed to release a broken version of their game and they admitted that Sony and MS gave them waivers on the after cdprojekt negotiated what they would fix at launch.
Even in the investor meeting cdprojekt was asked how were they allowed to release a broken version of their game and they admitted that Sony and MS gave them waivers on the after cdprojekt negotiated what they would fix at launch.
OP
OP
Not really. This is all that they said about certification.
Q: " And secondly, and related to that, at the certification stage – presumably, Microsoft and Sony always get games that still have bugs, and decide they're going to be ok – partly on the basis of discussions with you that there will be fixes. Have I understood that correctly? Or do you feel somehow that the certification process is kind of only one side or the other, and failed to identify just how underperforming the last-gen version was?"
A: "In terms of the certification process and the third parties – this is definitely on our side. I can only assume that they trusted that we're going to fix things upon release, and that obviously did not come together exactly as we had planned."
They admit that Sony and MS had reasonable trust in them to get it in good shape by Day 1 and that they failed to do so while taking the blame for the issues.
Interesting article!
My main takeaway from this however is that the cert process apperently takes way more issue with buttons being mislabeled and doesn't care at all the game was missing a very important epilepsy warning. Feels like something that should be part of the cert process imo.
My main takeaway from this however is that the cert process apperently takes way more issue with buttons being mislabeled and doesn't care at all the game was missing a very important epilepsy warning. Feels like something that should be part of the cert process imo.
I expanded on my post a little more, in case you wanted to update it.
But I do want to be clear, this was damn near a decade ago, but I don't think the cert process has changed that much, outside of getting more complicated as gaming moved to things like online infrastructures and microtransactions, and GaaS models.
That's almost exactly what happens. I was a QA tester for over a decade before moving into design.
In my experience, the QA team catches a lot of issues (I've worked on projects with tickets in the thousands). But ultimately, the dev and publishers make the call on whether or not the issue will be fixed by release. Each bug has a classification, and a priority. Z-fighting? Clipping? Those are low classification, low priority issues. Crashing the game? High classification, high priority. But if the repro rate is too low (ie, the tester can only get it to occur 15% of the time, and not even in the same spot), it may be marked as low priority, or "Will Not Fix.".
Ultimately, if it isn't a highly reproduceable, game breaking bug, it's probably going to be kicked down the list, if not waived altogether, and that's not on the QA team, who does their job, and does it well.
I admit I definitely get a little irked when I see the constant rhetoric "Did QA even test this thing?! hurr durr!" when the reality is that, yes, they most certainly did test it. Over, and over, and over, and over again, but it wasn't their call on whether or not the issue gets addressed.
I've been on both sides of the fence: the one submitting the bug, and the one fixing the bug, and neither are enviable. Sometimes your bug gets WNF'd. Sometimes you just don't have the time to fix the issue, or the fix could affect some other aspect of the game that's working correctly, and fixing the issue in this particular release is deemed too risky at the moment, so it gets kicked down the list until it's safe to poke around and get to the root of it.
It's why patches and updates can often take so long to come out: any bugs have to be carefully researched, isolated, and tested against the other systems in the game to make sure there are no unforeseen circumstances. I remember years and years ago, there was a bug that seemed pretty innocuous, but the fix broke something major, so the executive decision was made by the dev team (I was QA at the time), to let the innocuous bug slide.
Making video games is hard as fuck, and so many moving parts have to be coordinated behind the scenes to try and make the product as polished and smooth as possible, and sometimes you can't address everything, at least not right away, but the intention by the team and studio is to always address as much as they can, as fast as they can.
Last edited:
The more neutral rule is to just report the issue and not suggest how the fix should be done.
It's upto the <whoeveritgoesto> responsibility from either the art or dev side on how it should be changed and how it should be fixed.
Edit: I'd also mention that those who report issues shouldn't take their issues very very personally; like if an issue is shot down by a reason or another and its not a major issue, there shouldn't be a crusade to invade thoughts of the art/dev team to have those issues fixed. I have seen people get very personal over this, it shouldn't be.
Last edited:
Back when third party PSN games used to be released in EU months after NA, a common view seen online was that it was because SCEE was taking too long to translate the game into European languages.
Most people have no idea how any of this works.
QA --> find issue--> describe (reproduce) problem --> report bug
Now I bet that in the CDPR ticket system are more than enough bugs and issues that have been discovered but shipped anyway. It is a matter of severity and priority which is heavily influenced by outside factors (like a shipping date).
So blaming QA, DEV or their team leads is often the go-to response of many but it is not the cause of most issues. I worked in all three roles and whereas the team was usually quite aligned what needs to be fixed or done next - others preferred new features over bug fixes anyway ...
My job is very collaborative, so we allow our QA team to suggest things as well, but it's generally acknowledged that not all suggestions will be acted on, but we're more than open to the feedback. Not to mention that we often promote from within the studio, so if a QA tester is an aspiring game designer, it's encouraged to not only express that interest in wanting to get into design, but also feel free to offer constructive feedback. Hell, that's more or less how I got promoted from QA to design. I've definitely worked at places where some members of the Dev team looked at QA as being beneath them, figuratively and literally. It was super demoralizing and uncalled for.
It's often hard not to take your bugs seriously. After all, you're putting a lot of blood, sweat, tears, and often grueling hours, to do your job and report on issues, so when an issue gets dismissed, closed as "Will not Fix," or whatever, it definitely stings. How a developer responds to your ticket is also important. Like the dev in that post above sounds like a dick, and very unprofessional. I had a dev respond to one of my bugs like that at an old studio I worked at, and he got chewed out for it by the lead dev. I was new, eager to do a good job, and didn't realize that the issue I wrote a bug on had already been known, and marked as a "Will not fix." I don't remember specifically what his response was, but it was unnecessarily hostile.
Whenever I have to close out a bug, or WNF a bug, I try to make sure that my reasoning for doing so is worded in a way as to not make the QA tester that reported it feel like shit. It's a two way street.
Last edited:
As somebody that just moved into software testing I had no idea that it was so bad at a lot of places. Honestly, makes me grateful for the people I work with.
The whole trend of going gold on broken or subpar iterations of a game and then relying on day 1 patches was bound to blow up in people's faces at one point or another. It's good for developers to have that leeway, but it's a not a great for experience for players when the disc has to act like a key for a download of the 'proper' version of a game.
Aside from that, when platform holders allow this many "ifs" and "shoulds" into the process (while accepting pre-orders), consumers should have a robust (digital) refund system as a counter weight if they are unsatisfied with the quality or state of a game day 1 or whenever. That kind of accountability doesn't seem unreasonable to me. Telling people to wait for a theoretical future patch when their game is crashing and they want a refund is more than a little brazen.
So while CDPR is ultimately at fault for not making good on the promises they made, it shows to me that platform holders are both letting them get away with too much and not giving consumers enough avenues for recourse. They're not entirely scot-free here, imo; it's not a one-off whoopsie daisy that got us here and there's no way it couldn't happen again. A formalized refund process like on Epic and Steam will at least partly help offset the risks the production and distribution side is willing to take on (as opposed to the current "OK but just this one time" arrangement that we also saw with No Man's Sky, for example). People are too quick to relieve Xbox and Sony of all accountability just because CDPR "should" have delivered and on the technicality that a game crashing regularly or being full of progress-halting bugs isn't that bad because cert is just to make sure it doesn't entirely nuke your hardware. Not a particularly high bar if you ask me.
tl;dr give us a formal refund process, you cowards 🤡
Aside from that, when platform holders allow this many "ifs" and "shoulds" into the process (while accepting pre-orders), consumers should have a robust (digital) refund system as a counter weight if they are unsatisfied with the quality or state of a game day 1 or whenever. That kind of accountability doesn't seem unreasonable to me. Telling people to wait for a theoretical future patch when their game is crashing and they want a refund is more than a little brazen.
So while CDPR is ultimately at fault for not making good on the promises they made, it shows to me that platform holders are both letting them get away with too much and not giving consumers enough avenues for recourse. They're not entirely scot-free here, imo; it's not a one-off whoopsie daisy that got us here and there's no way it couldn't happen again. A formalized refund process like on Epic and Steam will at least partly help offset the risks the production and distribution side is willing to take on (as opposed to the current "OK but just this one time" arrangement that we also saw with No Man's Sky, for example). People are too quick to relieve Xbox and Sony of all accountability just because CDPR "should" have delivered and on the technicality that a game crashing regularly or being full of progress-halting bugs isn't that bad because cert is just to make sure it doesn't entirely nuke your hardware. Not a particularly high bar if you ask me.
tl;dr give us a formal refund process, you cowards 🤡
Last edited:
In my experience a lot of the "bad" collaboration came from the huge difference in skill-set between QA and dev that was mainly due to wrong hiring requirements. The manual tester that "clicks through the application". Nowadays someone in software testing will write code for frontend automation test cases, take care of automatic deployment pipelines, has an ISTQB certification and write the test concept and so on.
That's exactly why I say "not to shift the blame on one person". The problem ultimately comes down to management but a Producer has much more responsibility over managing this process with QA than a programmer.
I think things have changed a bit since I worked in certification for a platform holder QA between 2006 and 2013.
It was absolutely within our duties to report every kind of bug from small graphical glitches to system hangs. Bugs were assigned a points value based on severity, and if a title accrued enough points, it would fail. My job was in a department called Format QA. The entire certification process involved several different departments though. As the name implies, the TRC dept dealt with 'technical requirement checklist' issues. This is what people seem to be referring to when referring to certification as a whole. There are also network departments and compatibility, which test games within their respective domains.
From what I hear from friends who still work there, what remained of Format QA was rolled into TRC and cert became what it is today.
This is all relatively recent though, and I guess my point is that a game like Cyberpunk could absolutely have failed cert based on graphical glitches and crashes had it released back in the PS3 days.
It was absolutely within our duties to report every kind of bug from small graphical glitches to system hangs. Bugs were assigned a points value based on severity, and if a title accrued enough points, it would fail. My job was in a department called Format QA. The entire certification process involved several different departments though. As the name implies, the TRC dept dealt with 'technical requirement checklist' issues. This is what people seem to be referring to when referring to certification as a whole. There are also network departments and compatibility, which test games within their respective domains.
From what I hear from friends who still work there, what remained of Format QA was rolled into TRC and cert became what it is today.
This is all relatively recent though, and I guess my point is that a game like Cyberpunk could absolutely have failed cert based on graphical glitches and crashes had it released back in the PS3 days.
But I am the manual tester that "clicks through the application" (for now at least). I mean, my company (government adjacent here in Austria) paid for my ISTQB certification which I finished on monday and will put a considerable amount of time and money into developing my skills like test automation and so on, but honestly, I know all the devs I work with pretty well by now. Also had a few long video conference sessions with our project manager where we just talked, unrelated to work.
Maybe it also helps that all our bug tickets go through 2 intermediaries (test manager and project manager) but I have never heard anything of what people described in this thread, fortunately. They just seem happy that someone found stuff and it didn't go live. Hell, at my old job the devs were just straight up happy that they didn't have to test it themselves anymore.
If there is an opportunity for collaboration and the communication amicable, its the best environment. Very dependent on leadership and process.
Sometimes the dissociated nature of outsourcing affects this and so does closer deliveries for commitments, a neutral communication keeps people from going over the edge of frustration (hopefully on both the sides).
Note: For others not in the industry, issues/bugs are also a form of communication between tester and the development teams. It's not exclusive to just opening IM's, Mails or Phone calls.
Funny coincidence - I am from Austria.
I did the ISQTB certification as well back in the days. For a start manual testing is fine - the domain knowledge is essential to find issues. However, the more complex software becomes and depending on deadlines, new code, ... automation is mandatory *if* possible. I also never had such experiences personally but I noticed it in other departments / with other colleagues. The reason was that the discussion level between the two involved parties was too big.
There is a long road from "This is broken. Fix it" to "Here is the stack-trace. I already can tell that the following reasons did not cause the issue." to even more details and an understanding of the project.
Sure, cert isn't responsible for testing the quality of the game. And it shouldn't be. But it should be responsible for testing the technical quality of the game. If your game crashes at least once per hour, or if your game fails to run at a stable 30fps minimum, your game is fucking broken and you should not be allowed to release it.
If not running at a stable 30fps at minimum was actually made a reason for failing cert, a whole lotta games would have failed to see release... especially back on the PS3.
As someone that was, how to put it, "extremely involved" in cert process for one major platform holder, this post and the thread from Rami are spot on.
I worked on The Sims 3 (X360/PS3) and Sims Medieval. For Sims Medieval they asked us for suggestions but I think they didn't even read them.
He's right.
Unless Cyberpunk is crashing like crazy or has progression blockers. Cert absolutely tests for crashes and progression blockers. If they can reproduce it 100%, then it could fail your title at cert
Cert will sometimes (often?) Waive crashes or critical cert issues on big titles (like EA games...or Cyberpunk), that release during the Christmas period, in order to allow them to make Christmas, or make a crucial launch window, so long as the company promises to fix the issues in a patch.
I'm curious how bad Cyberpunk actually is. Those console reviews look rough, but it's hard to tell if that's just last-gen consoles, or of Series X and PS5 are just as bad?
Unless Cyberpunk is crashing like crazy or has progression blockers. Cert absolutely tests for crashes and progression blockers. If they can reproduce it 100%, then it could fail your title at cert
Cert will sometimes (often?) Waive crashes or critical cert issues on big titles (like EA games...or Cyberpunk), that release during the Christmas period, in order to allow them to make Christmas, or make a crucial launch window, so long as the company promises to fix the issues in a patch.
I'm curious how bad Cyberpunk actually is. Those console reviews look rough, but it's hard to tell if that's just last-gen consoles, or of Series X and PS5 are just as bad?
man what the fuck
i've been looking at making a career shift into a producer or PM role at a studio and seeing shit like this makes my blood boil. if that ever does happen, I wonder if PMs or producers have any clout to push back on programmers.
How so? If they waived it all (as in knowing about the issue before the launch of the game, but got the certification passed anyway), wouldnt that already be fixed in a day 1 patch? Such serious bug would have to be fixed fast after all, so i doubt that they knew about this issue before. Once the issue got known after the launch of the game, it didnt take that long before it was fixed, unless i'm mistaken. Didnt the crash happen more randomly by the way, or did it happen in the same spot everytime? But the reason for the crash could maybe be related to something else that isnt a part of the certification checklist, but i'm not sure.
Last edited:
As far as I know, it doesn't unless you fail cert multiple times. I'll tell you for sure in a few months hopefully. :)
It breaks my heart to hear how poorly QA is treated in big companies. Every AAA developer should work on indie games first and be forced to do their own QA; hopefully that would make them appreciated and respect it more.
I wonder why Cold War isn't getting any flak when it has crashed and brick Ps5s.
The game had no QA and has even prevented people playing on Xbox and PC
The game had no QA and has even prevented people playing on Xbox and PC
And to add another layer: Developers hate the cost associated with QA, which they find to be excessive. They will often try to cut the number of testers employed to even beyond the minimum needed to properly test a game. And whenever they'll come across a bug that wasn't found by that poor understaffed QA team, they'll blame the QA team for it.
Once I was on a project where the Lead QA was ringing alarm bells about needing more testers to properly test the game, that they couldn't keep up with the developer's pace and soon would become flooded. The game producer said no and told them to do more overtime, as if they weren't doing that in excess already. Eventually it became clear that they weren't going to make it, and the producer finally caved in, but it was too late to run the normal recruitment/training process to find enough people in such a short time, so they turned to in-sourcing testers from an external QA firm, which ended up costing even more.
Afterwards, who do you think had to go to the company's international HQ to explain this fuck up and essentially defend themselves for what had happened? The game producer? Hahaha nope. They sent the Lead QA. They literally threw him under the bus.
Honestly wondering since i'm not familiar with it, but doesnt most people who work with big titled production already have experience from smaller productions first (like indie games)?
I wonder if its more of an individual thing (which can also technically happen in smaller production, at least to a certain degree), that some developers feel that their work is more important, and maybe they take some feedback from the QA team as some form of criticism of their work, and get kinda annoyed/angry as a result of that. But i'm not sure.
That sucks, but on the other hand, wouldnt this give the best chance for the Lead QA to explain how the situation really was? If they had sent the producer, couldnt he/she have blamed it on the QA team then without them (the QA team) having a bigger chance to explain the situation?
And out of curiousity, do you know what happened aftwards by the way? =)
Yeah I always thought cert has nothing to do with bugs, glitches or performance.
Otherwise we wouldn't have gotten gems like:
Otherwise we wouldn't have gotten gems like:
I worked on a MMO-like game and my QA team (outsourced) was 4 people (including me). We should have been 15-20 easy, at minimum, but never really ramped up beyond that.
The devs would send us daily checklists (with over 50 individual checks) that made no sense either, including performance checks, and I was pushing back constantly on all of that because it wasn't fair to my team until I left the project.
The way QA gets treated sometimes, especially external, is super gross. Even the companies that offer these external QA services are gross as the suppress wages, engage in nepotism beyond belief and a whole bunch of other horror stories.
I don't think so. Indie games for the most part weren't a thing until a decade ago, and indie devs tend to remain indie (at most joining a larger indie studio, if that). As an indie myself, I can't imagine working in AAA games. I'd rather go back to work an office job, frankly.
A more realistic path for AAA devs is probably being hired as a junior dev without experience, or, ironically, QA, and rising through the ranks as they accumulate experience.
Interestingly, what does happen is the opposite. Several prominent indie devs have a background working for big publishers, then quitting to do their own stuff.
As someone who works in software QA (Not for strictly "games"), my heart goes out to people testing. Identifying a big backlog of issues that doesn't get addressed is a common occurrence. Triage and making hard calls on what does and doesn't get fixed is a challenge, but a necessary one for any project. But what the team at CDPR no doubt felt (Or in past Bethesda games for example) seeing all those bugs going out the door must have been crushing. I likewise wouldn't be shocked if they're comparatively greatly understaffed and were specifically told not to spend a lot of time on last gen testing.
My company thankfully has a lot more respect for QA and we're highly collaborative in the overall development and design process. But even then, it's easy to have the finger pointed in your direction when something goes wrong.
My company thankfully has a lot more respect for QA and we're highly collaborative in the overall development and design process. But even then, it's easy to have the finger pointed in your direction when something goes wrong.