• Anno 1800
  • DevBlog

DevBlog: QA Detectives

Hi guys, my name is Dorina and I am in charge of the Anno 1800 QA department. I have been working in games for 13 years by now, starting as a student assistant on Anno 1701. A few years later, I got a permanent development testing position and then worked on Anno 2070.
It is a great honor to lead the QA team for Anno 1800 and I hope that this blog will give you an interesting look on the diversified tasks of our Quality Assurance team.

When it comes to game testing and QA, most gamers out there already seem to have a set opinion about that matter, and often make their voice heard. If you are able to identify bugs in your favorite game, then it surely can’t be that hard for the developers to fix the issue, right?
Today, we want to shine a light on the Anno 1800 QA team and give you a look behind the scenes of our daily work, in order to paint a clearer picture of game testing and the complex nature of game development.

QA tropes – playing games, all day long
There is a common trope that working in QA means playing games all day long, and while getting your hands on the game is surely part of the deal, reality is quite different. In fact, spending the day playing the game to check its overall shape and functionalities happens only occasionally. Our day-to-day work consists of specific tasks, where we have to test a certain feature of the game to evaluate its quality or to investigate reported issues. A complex matter, as reproducing a situation can be a demanding task and the causes and effects might often seem totally unrelated at first glance. We get an extensive features list to test for every new development milestone. To reach our goals up to the next big milestone, cooperation in our team and with the feature stakeholders (such as a game designer responsible for said feature) is crucial.

Our QA team’s desks are located in the middle of the studio, as we are a central focal point of production, being the team with a broad overview on the entire game. That allows us to build a strong connection with the other teams; you can almost say that QA is the watching eye of the production. People also might get a bit nervous when we spawn at their desk, as that usually means that their features are partly broken in some way.

The detective and his handbook
When we start investigating, our analytic skills, experience and knowledge about various development disciplines are our main weapons. While being on a case, the detailed design document (also referred to as DDD) is the QA detective’s handbook. While testing the game or analyzing issues, we always check back with the DDD if the game elements work as intended. As the DDD defines the function of every  element in the game, it allows us to create both kinds of test cases: the ones where we check if the game behaves as intended and those where we try to break the system on purpose. We will then compile all data in feedback reports and deliver those to the responsible developer and to the other stakeholders.

You all remember how we talked about the trade routes feature in last week’s DevBlog? The DDD describes that feature and all underlying rules of the system in detail, such as how the ships are being added to the routes, but also how menus should work and look, related audio elements, text display, button functionality and much more. When testing that feature, we have to work with the feature checklist to ensure that every single point is properly working. After a bug is fixed, we have to perform further checks to see if the fix breaks any other element of the game.

But it is not only about testing if every element works or if there are hidden bugs, we also help to test the balance of the game, from the production chains to the assessment of the difficulty of the battles. We always have to ask ourselves: is it working as intended and is the feature complex enough (or sometimes even too complex)?
A task rarely finished in one day, as it involves feedback reporting and ongoing discussions with the responsible developers. When the feature owner then applies changes based on that feedback, it means back to the testing process again.
But that is not all- another difference between the QA and QC’s (quality control) work is that we also check the projects globally. If we find out that a tool is not working properly or if there might possibly be a blocker in the production process, we will provide a report to the production team.

Bug hunting!

If we find a bug during the testing process, we will track it down in our bug-tracking tool and forward that report to the responsible game designer. As described before, that kicks off the testing and feedback loop until the issue is resolved.

Are you familiar with the saying that goes “No battle plan survives contact with the enemy”?
When testing and resolving bugs, there is always a probability that the fix breaks something else in the game. Most of the time, we do not know where a blocker is rooted so we have to find out what’s actually causing the issue first. Games are incredibly complex and there is one general rule: if we fix one bug, it might cause another.
When explaining that, I like to refer to the “butterfly effect”, where every flap of a wing can cause something unexpected and kick off a whole series of events. Therefore, before we release an update, we have to test the whole game again to verify that the fix is not breaking something different, which might not seem to be related to the initial issue at first.

This is a case-by-case process and the complex nature of a game, where bugs appear often during the development and we have to ensure that this doesn’t block anything in production process, as bug fixing can require hours of work from our coding team.

Make the game better
But there is another layer of our work which is not necessarily related to bug testing and fixing. When going through the DDD and observing the game, we might notice that something is missing or might cause gameplay issues based on our experience. If that occurs, we will provide a report including improvement suggestions of what might be missing or what could be changed, in order to improve or to balance the game’s content. The knowledge in our team is hereby of a deciding factor, as we are all different types of players and experience levels. That exact level of variety is important to be able to observe things from as many perspectives as possible.

How about you?
I hope that this blog gave you a good overview of the complex nature of game testing, as it affects every aspect of a game’s production across every development discipline.
Working in QA requires enthusiasm, passion and a deep understanding of the games in general, as well as the title you work on.

What your takeaway on the role of a professional game tester? Was the topic all too familiar, or did you get new information? I would also love to know if you have ever tested bugs in the past for games or other projects, maybe just as member of a community or even on a semi- or professional level?
It requires a passionate and dedicated team of experts to tackle the daily QA challenges of a complex strategy title like Anno 1800.

Comments

 

 

14 Comments
  1. S Soulridder February 23, 2018

    Interesting how the testing process of Anno works.
    I guess you are doing it a lot different than the community teams (mod development; Minecraft) I worked with where all Beta and Alpha testers could provide bug reports while the developers itself decided on how critical the issue is while also dismissing about 20% of all bug reports as the feature was intented that way, thought no one told the testers. ^^

    So yeah, I was a Beta tester on a few games and mods for games in the past and I would really have whished to have sometimes a design document laying around, especially with pathfinding and AI issues, but I’ve never tested those mods or games on a professional level.

  2. N Naviki2018 February 13, 2018

    Hello, my name is Estéfany. I am Industrial Design Engineering Student at Instituto Tencnológico de Costa Rica. I would like to be part of this, to learn. I know that it is not to play all day. I know how to do a report, but I don’t know if I must to life in your country to help with the department. I would like to send my portfolio. Where can I send my information?? It is a requirement to be graduated to help in the department??

    • N Naviki2018 February 13, 2018

      ** if I must live in your country…
      Excuse me

    • B BB_CR February 16, 2018

      Hey Estéfany, thanks for your comment. It depends- if you want to participate in external testing like our online tests, you can do those from home.

      To work full-time at Ubisoft, it is always on-site in the various studios. While it is not Anno, maybe some of our studios in the NCSA region may have openings of interest to you: https://www.ubisoft.com/en-us/careers/search.aspx

      Best,
      Marcel

  3. r ruuti0 February 9, 2018

    I have one question: I heard that people who work in Ubisoft usually have a Degree (University Degree), but what kind of education people who work in Quality Assurance team exactly have?

    Is it something related to computers (such as Compiter Science) or something else or does people come from different backrounds to work in your QA team?

    Thank you in advance for answering my question!

  4. r ruuti0 February 9, 2018

    For first thank you again for great article!

    I learn a lot of more what Quality Assurance mean!

    Professional game tester sounds like interesting profession!

    Some of game testing I knew, because I have done it myself on smaller scale on smaller games that me and my friends made, but I also learned something new such how big game companies work together with QA and other parts of company.

    I have myself tested bugs in games in past, those games were made by my friends and it was fun. I have also made small games myself and tested (tried to find bugs etc) them as well.

  5. d diddle783 February 8, 2018

    thanks for the behind scene and explanation of your job. I do know that a bug in a game is not an easy fix like some players may think and sometimes it takes a lot of time to fix..therefor respect to the enitire team behind the game, because of al your passion. dedication towards the game we as players can enjoy a game that will be good once it is released

  6. B Boldzmann February 8, 2018

    I know your work well. I have been working for QA for almost 5 years. My task is to check localizations. Usually it’s 9-13 languages. If I check the text, I need it to be correct in all languages. In addition, it should be placed in the elements of the interface. In addition,
    there are other mistakes. We divide them by priority into 4 types.

    Critical – Critical error, malfunctioning logic, hole in security system, the problem is the inoperative part games, without the ability to bypass it, using other input points. Easily will be found by the user, if not corrected. Critical all bugs that prevent the release of the product are cause negative consequences for business and the user.
    For example: The game stably falls when clicking on the building menu.

    High – A significant error, part of the logic does not work correctly. There are possibility to work with the tested function, using other input points. This type is used if the bug is serious enough to pay attention to it. If it can not be fixed, it can cause tangible inconveniences for users or for business. For example: When building a
    building for building materials, the amount of materials is duplicated to other buildings.

    Medium – Minor bug that does not violate the logic of the tested part for example, the interface problem. Bugs of this type will not cause serious Harm, if will be missed in the release. Assigned to bugs that you need to fix it, but not in the first place. For example: the icon of the market partially closes the icon of the market square.

    Low – Trivial mistake, not concerning logic, is bad reproducible problem, unobtrusive in the interface, non-critical the problem of third-party libraries or services. Not serious influence on the perception of the product by the user. This type means that The bug,
    rather, has the character of a wish, does not create a significant inconvenience in interacting with the game. For example: there is no animation for opening the construction window.

    We work with the application in which we create tasks for each error. Specialists have access to the task. So we jointly solve it. For a serious mistake, you need to find steps to repeat it. If the error is visible then to the task we add a screenshot. For some tasks, you need to add crush file and(or) log file. In some cases, other files are added.

    I played a lot of different Anno. And I got a few low errors. When I played in 1404, it rarely happened that the game crashed during autosave. But it did not in any way affect my love for the game. It’s quite a complicated game. You must be serious specialists as you work on such a complex game. Thank you for sharing, I wish good luck in your work!

    •   Bastian Thun February 9, 2018

      Thanks for sharing your experience in that splendid post with our community. We also categorize bugs to identify if it the issue is a minor or major issue and we also have to estimate the time to fix it. It’s about priorities, it makes sense that you tackle a major bug before you fix a minor one which takes a lot of development time while there is always time for simple bugfixes even if you have a list of major issues.

  7. p palemale53 February 8, 2018

    I have been a developer for most of my life, and am only too well aware of the difficulties you face. I cannot imagine the number of detailed test cases you need to have under control. In my experience as a developer, any feature needs to have more than a few, perhaps dozens of tests.

    Then every so often everything needs testing, just to catch those butterfly wing flaps. I cannot believe that this testing is done with actual game play. Much of this testing needs to be automatic and fast. As a developer, I would love my tests to be done in a few seconds, to make the testing part of my development. As a team manager, I would like a comprehensive test suite to run overnight at worst, so that my people are not kept waiting. At best I would like integration testing to be done no more than ten or fifteen minutes so that the tests can be run whenever a developer commits his work. The easiest bugs to fix are those that have most recently appeared.

    The sort of thing I would do would be to expose the inner workings of the game in their finest detail. The tests would consist of a set up phase that puts the feature into its desired state, a bit of code to exercise the feature being tested, then one or more tests to ensure that the results are what they should be. In a complicated situation, I would repeat this exercise and probe step as often as required. Ideally I would then throw away the game state to start the next set of tests, but it may be desirable to return to an earlier state instead.

    This is fine at the feature level, when you have the DDD to guide you. When you have bugs in game play your detective skills demand all the evidence! In the worst case you would have a memory dump of the game and a trace of recent player actions – muddy footprints! From this you need to infer what the situation was before things went wrong, and what actions lead to the crime! Your case report must be stripped of everything unrelated to the problem. It must be possible for the developer, as your forensic assistant, to set up the detailed circumstances of the crime, repeat the fatal step, and show with a failing test what went wrong. You now have a repeatable test that can be added to your automatic tests.

    Is this a “who done it” fantasy on my part?

    I cannot imagine how your job would be possible without something like this.

    •   Bastian Thun February 9, 2018

      Thanks for the detailed post! We make surely use of automated testing, such as conflicts which appear when compiling a new development build. While we focused on the actual investigative part of the job, every new build generates a new list of errors which then become a part of the next milestone goal.

  8. A AmpeImann February 8, 2018

    Interesting to know you QA people can submit feedback to the DDD. One question I have, there’s a lot of focus these days on rapid development and agile methodology. Are all core concepts of the the game written in documentation before any developers start coding? Does the development cycle of an Anno game look a little more like traditional development?

    On and personal note it’s pretty cool that you started your career on an Anno game and have continued on working on Anno games. Looks like you and other members love Anno as much as some of us players do!

  9. M MMateos19 February 8, 2018

    Unfortunately, I haven’t tested any game thoroughly, basically because it’s hard to get a beta of any kind of game since a lot of people apply (and other factors that I suppose may vary depending on the company), but I would really love to have the chance of testing this game and provide feedback to you guys. Personally, I think the Anno franchise is the best building game because of the chain productions of the goods. That’s the strongest point of the game for me, since you can’t imagine having a population that doesn’t satisfy its needs, and the Anno approach to this problem is the closest one to reality, starting with a basic element such fish and finishing with cameras for example in Anno 2070.
    Now, after selling myself, this blog is very interesting because of several reasons:

    Every game has a community, which can be more or less friendly. In my understanding, the best a company can do is to release the game with the less quantity of bugs possible, since I’ve seen a lot of games receive bad score because of this, and that affects enormously the opinion of the people who were wondering about buying that game or not. Now, as a programmer myself, the gamers don’t see that behind a game there are millions of lines of code and that threads are really hard to dominate, giving an unexpected behavior in any moment, since if you miss an execution case that you did not code that bug can affect the whole game as stated in the article, but that bug can be resolved! That’s the main problem of communities. Instead of giving supportive feedback, they go into a rage, blaming the game and their developers (of course, this kind of players are much less that the ones that are supportive), when with a bug report and time it could be solved “easily”.
    That’s because this “Anno Union” webpage is so amazing, because the community can go into that rage before the game is released. I’ve seen very mean comments in past posts like in the multisession topic, so something can be done about this before the game release. If I were part of the development group of Blue Byte, I would post more blogs of this kind, explaning the problems that a programmer face when developing an Anno game and the solutions and approaches to that problem.

    In summary, keep with the amazing job you are doing developing this game and the feedback you are providing to the players.

  10. i iruet February 8, 2018

    Nice pics ^^

    This blog reminds me of that video that Chris linked some time ago 🙂

Updates

View all News