In the automated test environment E2E*, many technologies have been released for development over the years, both by the Flutter team itself and by external developer communities who have recognised the potential of this software development tool.

Today we want to take a look at a technology originating from the developers, Maestro.

*(End To End, summarising the path from point X to point Y of the application as QA from the user’s point of view).

About automatic testing in Flutter

To decide whether to opt for one technology or another when it comes to automating our application’s test cases, we first need to understand what exactly “automated flutter tests” are.

With a focus on “end to end”, Flutter provides the flutter_test and bdd_widget_test libraries. These are tools that are placed inside the application we develop and have full access to the different widgets, pages and lines of code and give us great control over everything that happens inside the application and are the best solution for unit testing in Flutter.

They also offer compatibility with Cucumber, which makes our tests much more readable for non-technical users thanks to the Gherkin language offered there.

Although all this is offered, one major point must still be taken into account. And that is the extensive knowledge that the tester needs about Flutter, the libraries mentioned and Cucumber. This is usually the biggest mountain to climb in terms of QA in our project.

And this is exactly where Maestro comes into play.

Maestro, effectiveness in simplicity

Written on the basis of Appium, Espresso, UIAutomator and XCTest, among others, the so-called “flow” (the definition of the user’s steps in the application) offers test options that can be executed very easily and immediately anywhere, as they are not part of the project but run externally..

The execution takes place via a .yaml format (a language that is easy to read and write even for people without extensive knowledge of technical terms).

It is designed so that the available commands not only make it easy to recognise each emulator played on the screen without having to link anything, but also so that the various widgets required to complete the desired test route can be found quickly.

To compensate for the lack of compatibility with the POM and Cucumber model, Maestro offers the possibility to organise a structure of nested executions called “nested flows”. With the help of the “runFlow” command, which enables the separate execution of Maestro files when nested correctly, the reuse of code and the ability to organise an easily understandable and manageable structure is made considerably simpler.

Detailed information about Maestro, its advantages and disadvantages

1. Simplicity is too simple

When we look at Maestro, it is immediately recognisable that we have already seen everything, which is both good and bad. We will return to this topic in the next section, as it is perhaps the biggest potential turning point.

Features such as the correct implementation of a Page Object Model, Cucucmber, database connectivity and other advanced additions are not possible with Maestro, at least not in the same way as offered by the other systems.

It is therefore essential to prioritize the complexity of the application to be worked on before choosing the tool, which is very powerful and with very good projection, but also relatively new.

2. Human language

Maestro is quick to learn and thanks to the use of the .yaml language, no advanced knowledge is required. It also offers the ability to execute commands with Javascript, which provides an interesting additional level of depth.

This simple code would already allow us to open an application on a device and simulate a keystroke.

This would be much more complex in the libraries mentioned above.

3. Variety of environments

Not only does it provide assistance with the portability of tests as it is not part of the base project, but the tool recognises both connected devices and simulated emulators from Android Studio (Android devices) and XCode (IOS devices).

In return for this convenience, Maestro loses tools on the tester’s side, as it cannot access things that are available in the base project.

Maestro at WATA Factory

Maestro is a tool whose usefulness and future prediction has been recognised by WATA Factory. The simplicity of test development is something not seen in other tools, and even if the tool is not used in some projects due to their complexity, it is always at the centre of discussions about automated testing in future Flutter applications.

The post The future of automated testing E2E in Flutter: Maestro appeared first on WATA Factory.

When it comes to defining the software quality and testing basics of a project, especially for web applications, deciding which technologies to use to automate our tests becomes a real challenge. This is even more true when we have to choose a technology to test our front-end.

If we focus on E2E (end-to-end) testing, simulating the behaviour and interaction of a real user by testing the user interface of our project, we find well-known technologies like Selenium, Protractor or Playwright.

But if there is one testing framework that is becoming more and more popular thanks to its ease of use and its advantages over the previous ones, it is Cypress.

What is Cypress?

Cypress is an open source framework for automatic front-end testing of JavaScript-based web applications. Its use is not limited to E2E tests, but also allows the creation of integration or unit tests.

Since it has its own assertion libraries, custom commands and other interesting features (e.g. a dashboard for debugging and running our tests), Cypress has proven to be a versatile framework adapted to the characteristics of the modern web.

All of this is based on an architecture that was built from the ground up: You don’t have to start with Selenium.

This allows us to work with a framework that does not require the installation of external tools and libraries to start building our tests.

Why Cypress?

Due to the development of web applications and the emergence of current and modern JavaScript-based frameworks such as Angular, Vue or React, E2E testing frameworks have also evolved. Considering the versatility and ease of use that Cypress offers, the popularity it has gained since its development is to be expected.

Another interesting feature of Cypress is that this framework runs the logic of our tests in the same execution cycle as the application. Cypress runs a NodeJS process that reacts to the events of our application in real time, without us having to use WebDriver to run our tests.

Cypress also makes it easier for us to develop tests, as you only need basic knowledge of JavaScript to get started. Assertions from other well-known test frameworks such as Chai and Mocha are also used.

Another important reason for using Cypress is, of course, the dashboard. A graphical interface where we can follow the process and execution of our tests in real time, as if we were in the place of the user, so that we can debug and have a direct overview of the errors that occur in our tests.

Relevant features of Cypress

Cypress has many features that are advantageous over other frameworks and that can help us decide to use it as an E2E testing tool for web projects.

1. current state of the framework and learning curve

Cypress is a modern and up-to-date all-in-one framework with good documentation, an active community and continuous updates.

It offers a quick setup of the framework in our project, as installation is fast and very little configuration is required. The tests are written in JavaScript and the Cypress library has descriptive and concise commands.

All this contributes to a flat learning curve for both the development team and the QA team.

2. Integration (or not) with other technologies.

Cypress does not require integration with Selenium because the architecture is built from scratch. In general, it does not need to integrate with other external libraries and frameworks, but we can do so if we want to extend the functionality of the framework with well-known technologies such as Mocha, Chai or jQuery.

It also works with any JavaScript-based library or framework that runs in a browser.

3. interaction with browsers

With Cypress, there is no need to install a browser as the headless browser (no graphical interface) allows for faster and less time-consuming execution. This would eliminate the need to install a browser to run the E2E tests through continuous integration.

On the other hand, we do not need to run a WebDriver in Cypress, as our tests are executed within a browser by a NodeJS process.

4. automatic wait times

When using Cypress, there is no need to insert arbitrary wait times between commands and assertions as this is done automatically, so there is no need to use wait and async in our tests.

5. useful and simple dashboard

Cypress has an easy to use and intuitive dashboard that we can interact with between each test step.

The dashboard provides us with dynamism as there is no need to restart the tests. Changes are automatically executed and are immediately reflected as soon as we save the progress of our file.

6. readable and versatile reports

The results of the console and dashboard tests in Cypress are very descriptive. In addition, there are many options for reporting the results of our tests, as well as the ability to easily take screenshots and record videos.

Cypress at WATA FACTORY

At WATA Factory, we focus on ensuring the quality of our software. Therefore, we consider it important to study well the testing tools we use in our projects, and we believe that the inclusion of Cypress is an up-to-date and optimal choice for testing our JavaScript-based web applications.

The post E2E test automation with Cypress: an elegant and affordable solution appeared first on WATA Factory.

In this article we will talk about Codeception, one of the tools we use for automated testing in our PHP implemented web projects.

What is automated testing?

When we talk about automated testing in the field of software testing, we want to achieve that manual tests can be executed unattended, with the help of a tool that automatically performs and improves the process, but without replacing 100% of the benefits of manual testing. We already reported on this in an earlier article.

If we can design the tests to be run by a computer, then we achieve continuity of testing while evolvingthe application.

Why is automated testing interesting for us?

At WATA Factory, we saw the need to implement this type of testing as the development of our projects required new functionalities to be tested along with the already existing ones in order to continue ensuring the quality of our products.

Automating tests brings a number of advantages that help to increase the quality level of the product and reduce its cost, as the repetition of tests does not require a large time investment in the phases of improvement or development of the product.

Some of the benefits we have achieved at WATA Factory through the implementation of automated testing are:

• Automatic verification of compliance with the defined specifications using BDD (Behaviour Driven Development). One of the big advantages of this is the use of a common language between the client and the development team to confirm that the behaviour of the code is correct from the user’s point of view.
• Reduction in the number of bugs, as we ensure that both new and old functionality continues to work correctly.
• Saving time in testing, as we avoid repetitive tasks and ensure that changes in our programme do not affect other parts of the system.
• Creating solid documentation.
• Making teamwork easier and safer.

To start developing automated testing, we researched various testing frameworks and came across Codeception, which combined everything we needed into a single tool.

What is Codeception and how can it be installed?

Codeception is a PHP test framework that aims to create readable tests and describe what is happening from the user’s point of view. It allows us to perform acceptance, functional and unit tests.

The key to Codeception is its actors. Codeception understands tests as processes that are initiated by a person. In summary, a UnitTester initiates actions and tests the code, a FunctionalTester tests the application as a whole, and an AcceptanceTester interacts with the interface.

Each actor executes so-called steps, which are individual instructions within a scenario. The instructions that an actor can generate are grouped into modules: PhpBrowser, WebDriver, DB, FileSystem, etc.

For the installation we follow the steps described in the official documentation.

During installation, a folder called tests is created and within this folder a basic structure for coding the different types of tests is created.

As an example, we will test the classic login. Although there are more scenarios in this test, we will only show some of them in order not to go too far.

To create the test, run the following command:

php vendor/bin/codecept g:feature acceptance Login

This creates a ‘.feature‘ file where we write our user story in a formal language called Gherkin, with the special feature that we can run these scenarios as automated tests.

Feature: Login page
Check that the user can access correctly with correct credentials, in other case a warning text is shown.
 
Scenario: User inserts empty password and empty username
   Given I am on login page
   When I fill fields with empty value
     And I click on login button
   Then I see the entered user data are not correct
 
Scenario: Successful access with valid credencials
   Given I am on login page
   When I fill the fields with correct credencials
     And I click on login button
   Then I see Welcome Dummy Name

Scenarios are written step by step using GIVEN-WHEN-THEN criteria.

• The GIVEN part is the precondition for the test Where to start from.
• The WHEN part is the specific behaviour, i.e. the actions we need.
• And finally THEN, the part where we get the expected result.

Defining the tests

Our next step is to define and convert the feature file into a valid test. It is defined in the AcceptanceTester file.

To get the necessary steps, the following command is executed:

php vendor/bin/codecept gherkin:snippets acceptance

Since we are in the Acceptance Suite, we use WebDriver as a module to implement, which allows us to interact with the browser. This means that we can use its methods within the tester file, just as we can with the script tests, with

$I->

One can use various methods that have already been developed (e.g. amOnPage or click). Each step of the Gherkin scenario is extended by the methods defined in Codeception.

We now show how this can be implemented in our case:

/**
 * @Given I am on login page
 */
public function iAmOnLoginPage()
{
 $I->amOnPage('/login');
}

/**
 * @When I fill fields with empty value
 */
public function iFillFieldsWithEmptyValue()
{
 $I->fillField('#username','');
 $I->fillField('#password','');
}

/**
 * @When I fill the fields with correct credencials
 */
public function iFillTheFieldsWithCorrectCredencials()
{
 $I->fillField('#username','correctUsername');
 $I->fillField('#password','correctPassword');
}

/**
 * @When I click on login button
 */
public function iClickOnLoginButton()
{
 $I->click('Login');
}

/**
 * @Then I see the entered user data are not correct
 */
public function iSeeTheEnteredUserDataAreNotCorrect()
{
 $I->see('the entered user data are not correct');
}

/**
 * @Then I see Welcome Dummy Name
 */
public function iSeeWelcomeDummyName()
{
 $I->see('Welcome Dummy Name,');
}

Finally, we run our .feature file with the following command:

php vendor/bin/codecept run acceptance Login.feature --steps 

to see that Given/When/Then is expanded with substeps and shows the expected result when executed.

Acceptance Tests (2)
-------------------------------------------------------------
Login page: User inserts empty password and empty username Signature: User inserts empty password and empty username
Test: App/Tests/surveys/Login.feature:User inserts empty password and empty username

Scenario -- 
Check that the user can access correctly with correct credentials, in other case a warning text is shown.

Given i am on login page
 I am on url "https://###########/login"
When i fill fields with empty value
 I fill field "#username"," "
 I fill field "#password"," "
And i click on login button 
 I click "LOGIN"
 I wait 5
Then i see the entered user data are not correct 
 I see "the entered user data are not correct" 
PASSED 

Login page: User inserts proper information - Successful access with valid credencials 
Signature: Successful access with valid credencials 
Test: App/Tests/surveys/articulo_test.feature: Successful access with valid credencials

Scenario --
Check that the user can access correctly with correct credentials, in other case a warning text is shown.

Given i am on login page 
 I am on url "https://########/login"
When i fill the fields with correct credencials 
 I fill field "#username","correctUsername"
 I fill field "#password","correctPassword"
And i click on login button 
 I click "LOGIN"
 I wait 5
Then i see welcome dummy name 
 I see "Welcome Dummy Name,"
PASSED 
-------------------------------------------------------------
Time: 19.57 seconds, Memory: 18.00 MB OK (2 tests, 2 assertions)

This tool offers us many advantages, it also has an application known as Webception where the client can run all the tests from the browser and thus be able to track the status of the application.

The post Introduction to automated testing with Codeception in PHP projects appeared first on WATA Factory.

So we can check whether the development process, the architecture used or the algorithms employed follow a suitable structure, a pattern that allows us to maintain the product easily.

What is SonarQube?

SonarQube is one of the most used tools to check codes, detect bugs, vulnerabilities and other problems in our project. It allows the analysis of a code written in the most common programming languages (Java, PHP, JavaScript, C#, HTML, etc.).

SonarQube is integrated as another file in the project to be analysed, and if the pipeline in which the tasks are started is configured correctly, the code inspection is performed automaticallyevery time we make a change in any part of the project.

As you can see in the image, the typical scenario in which utility can best be understood is one with three clear phases:

• Development – The code is updated by the developers in the repository. Before they ask the SonarQube service to analyse the changes, they can get immediate feedback thanks to the SonarLint tool, which can be integrated into the IDEs.
• CI/CD – When the new changes are added to the repository, the Continuous Integration tools check and build the code and run the tests. Then the SonarQube scanner is launched to analyse the results of some of these tests and the code as such.
• SonarQube platform – Once the analysis of the project is completed, the results are stored in the platform and depending on the configured quality conditions, team members can be informed if they need to fix a defect.

For the various projects carried out at WATA Factory, we have opted for the Community-Edition, which is free and open source. However, there are also paid alternatives that facilitate the installation and maintenance of the SonarQube service by the same company. The community version can be installed in two ways:

• Local – The developer can set up the SonarQube service on their localhost to be able to analyse their code without having to host it on an external or remote server.
• Remote – The SonarQube service is hosted on a remote server that can be accessed using credentials generated by the service administrator.

Depending on the needs and size of the project (members, resources or services to be used), one can choose the most appropriate option.

General concepts

To understand a little more about the relevance of using this tool, we will detail the general concepts that appear on the platform:

1. Users and groups: As in any environment, we can define users that are managed in groups. Each of these users has a set of permissions that allow them to request the analysis of a project, validate false positives of the analysis performed or even cancel it.
2. Projects: To carry out the analysis of our code, we need to create a project on the platform with the necessary parameters that identify our software project. hese parameters should be specified in one way or another, depending on the language we use in our project. Within these projects we find the analysis of each project, which contains the following data:
3. Bugs: errors in the code that need to be fixed ASAP.
4. Vulnerabilities: Places in the code that are open to external attack and can threaten the integrity and security of the project.
5. Hotspots: Parts of the code that should be checked to avoid major problems, they do not necessarily compromise the security of the project.
6. Code smells: elements that make the code poorly readabla or difficult to maintain.
7. Coverage: From the reports of the executed unit tests, SonarQube imports the results and displays the coverage.
8. Duplicates: number of detected duplicated blocks, files and rows.
9. Total lines: Total number of lines of code in the project.
10. Languages: programming languages used in the project.
11. Current status: Failed/Passed, depending on the values set in the associated quality profile.
12. Tags: tags that have been assigned to the project.
13. Time of last analysis: record of each analysis performed.
14. Quality profiles: They depend directly on the conditions defined in the Quality Gates, which indicate the rules to be followed in each of the languages available in SonarQube. The conditions of the quality profiles reflect the limits of the minimum coverage, duplicated lines, safety index or maintainability index.

As we can see, the report and the information generated are very comprehensive, giving us a very accurate overview of the status of our project.

At WATA Factory, we have established SonarQube, another tool that enables code quality improvement and also indirect learning among developers. Because with each of the reports, one learns which bad practices to avoid and how to solve themwith the suggestions offered by the same tool, thanks to the rules defined in the Quality Gates. In future articles we will see how we configure our project to be able to analyse it automatically from a pipeline with SonarQube.

The post SonarQube: How to maintain the quality of your code during a CI/CD process appeared first on WATA Factory.

As we have already seen in a previous article, there are two different types within software testing: manual testing and automated testing.

In this article we explain the features and the advantages as well as disadvantages of automated testing in a software project. To develop the content of this article, I have extended my personal experiences with those of various colleagues in our team at WATA Factory.

We will show the most important aspects that one should definitely consider in order to make the path to success as easy and fast as possible.

Features of automatic testing

Automatic testing is the approach based on the processes developed by test automation tools. One of its main objectives is to improve the testing process of a software projectas a complement to the improvement of manual testing.

With the current state, automated testing can never replace 100% of the benefits that manual testing brings us. Automation only improves the testing process by providing benefits such as parallel testing, automated reporting, elimination of repetitive tasks for manual testers or reuse of test scenarios. But currently, the software testing of a project cannot be fully covered by automated testing alone.

In this type of testing, the responsibility of the process lies entirely with the selected tool and the scripts that the tester has designed for the Application Under Test(AUT). Unlike manual tests, automatic tests cannot be applied in every area. Above all, automatic application still leaves a lot to be desired, especially in visual and UI tests. There are projects in which better tools based on AI are developed every day, but which still do not manage to achieve all the desired goals.

With this data, we can identify the main advantages and disadvantages of automatic testing:

Advantages

In contrast to manual testing, automated testing allows us to parallelise many workflows. The planning of these workflows offers a multitude of possibilities to improve the quality of the product. With a higher level of detail, we could say that automated testing offers us the following:

• Speed: The execution time is shorter.
• Reliability: more permutations and paths can be covered in the AUT.
• Efficiency: more tests are performed in less time and AUT coverage is improved.
• The tests are automatically executed from the scripts.
• The tests can be reused in different scenarios.

Disadvantages

On the other hand, the use of this testing technique brings with it a number of implications. Among these, the most prominent is the need to have a more technically trained testing team to design the scripts. And as we have already mentioned, the limitations we have to consider in different areas, such as user testing. If we analyse these disadvantages, we can get the following list:

• The testers must have technical knowledgeto be able to implement the test scripts.
• Cannot be applied to all possible types of testing. For example, the visual test.
• If the scripts are not designed correctly, false negatives can be produced that reduce the reliability of the reports.
• We cannot automate aspects such as the degree of usability or how intuitive the AUT is. For that we need the manual test.
• Recommended for consistent and long-term projects, due to the technical investment to be made.

The use of automated testing is becoming more and more common in the various areas of software, due to the many advantages it offers. Because despite the possible disadvantages we face, the result of a good application is always positive.

However, based on all these details we have analysed, it is necessary that two very important aspects are considered by the automated tester: the selection of a good test automation tool and the appropriate design of the test scripts. In future posts we will discuss how we can meet these challenges and how we can implement this kind of testing in our projects.

The post Automated Testing: Features, advantages and disadvantages appeared first on WATA Factory.

It represents a complete parallel cycle within software development, and this cycle is called the Software Testing Life Cycle (STLC).

Within the STLC we can find two different types of tests: manual tests and automatic tests. In the case of manual tests, the activity is performed 100% by the tester. And in the case of automatic testing, only a certain proportion of these activities is performed by a tester, while the rest is done by automation tools.

Features of manual testing

Manual testing is the most widespread technique in the history of software development. It is the first to allow the application of tests at different levels: Unit tests, integration tests of components, user interface… But even if any type of test can be applied with manual testing, there are certain disadvantages. Let’s look at the features in detail:

Advantages:

• Manual testing allows any kind of test. In fact, only this category of tests can control quality at the user interface (or usability) level.
• It allows the analysis of more complex scenarios thanks to the ingenuity that the tester can develop during his work.
• The risk of finding a false negative result is very low due to the direct interaction with the system under test.
• The testers do not need to have any technical knowledge to perform the necessary tests.

Disadvantages:

• The implementation of the tasks is very slow due to the difficulty of some scenarios.
• The tester has to be creative, patient and equipped with initiative to find situations that put the product’s properties to the test.
• Completing tasks takes a lot of time.
• Very tedious due to the manual execution of all steps involved in each of the tests.
• It is difficult to quantify the level of test coverage we have with this type of test.

Implementation in practice

At WATA Factory we can look back on a long series of projectswhere we have implemented manual testing techniques to ensure and guarantee the quality level of our products.

Each project has different characteristics and the implementation of this requires an analysis of resources and requirements, that need to be reviewed. Therefore, the following steps are recommended when starting manual testing:

1. System analysis: definition of the functional and non-functional requirements to be tested.
2. Analysis of available resources: staff, equipment, time, etc.
3. Planning of the manual tests to be developed: types and execution times.
4. Carrying out the tests.
5. Documentation of the results obtained: Error reports, result reports and analysed requirements.

Many of these steps are common in the STLC process, but in the case of manual tests their application and execution is quite sequential.

This is why we are starting to use automated testing techniques in our projects, which will enable us to reduce the deficits of manual tests and thus increase the quality of our products.

The post Manual Testing: applying manual tests to a software project appeared first on WATA Factory.