Getting started with PyTest

PyTest, is an open source testing framework for Python. Its widely, used from simple to complex applications. Its rich features, made it the most popularly used Python framework for unit testing, although it can be used to test Database and UI related scenarios. This framework gives us flexibility to choose which tests to be executed or skipped through simple rules. It also allows parallel execution of tests which saves good amount of time for lengthy test suites.

Environment used: Python 3.7 in PyCharm

Install pytest : pip install pytest

Confirm the installation: pytest -h, which will display the help.

Now that we are good to start, below is the list of basic PyTest features covered here: -Naming Conventions -Adding Assertions -Basic Test -Fixtures -conftest.py -Parallel Testing

Naming Conventions:

PyTest has set some simple rules to follow to name tests and test files in order to get them executed.

Filenames: Filenames should start with test_ or end with _test to be identified by the framework to execute them by default.

To execute tests in a directory we use the command: C:\PyTestProject> pytest

If a directory contains files named as: test_add.py add_test.py demo.py

test_add.py and add_test.py are executed automatically as pytest command detects them as

test files.

We can also execute tests in a file, that do not follow the naming convention by explicitly specifying the filename after pytest command as, C:\PyTestProject>pytest demo.py

We can also use this way to run any one test file explicitly, C:\PyTestProject>pytest test_add.py

Test names: PyTest, only considers to execute the definitions that start with test. If a definition name do not start with test then, it’s not executed and there is no way to explicitly make it run. Hence, if you decide the definition to be executed by PyTest then, do not forget to prefix the name with test.

If a file contains the following definitions, which ones do you think will be considered as tests by PyTest:

test_name() testaddress() city_test() test_state() citytest() details()

Hope you got it right, the following tests are executed.

test_name() testaddress() test_state()

Adding Assertions:

Before we start with a basic test, let’s see the assert statement in PyTest.

def test_compare():
    assert 5==6

Here, the assert keyword compares the values and returns the result as True or False, in this case its False.

Output error shown as:

def test_compare():
> assert 5==6
E assert 5 == 6

Detailed assertions: Here we add 2 numbers, compare to a result and also mention the details with the following syntax when an assertion failure occurs.This way helps in debugging.

def test_add():
    a=2
    b=3
    assert a + b == 6, "Expected: {}, Result: {}".format(a+b,6)

Output error shown as:

def test_add():
a=2
b=3
> assert a + b == 6, “Expected: {}, Result: {}”.format(a+b,6)
E AssertionError: Expected: 5, Result: 6E assert (2 + 3) == 6

Another test that returns True.

def test_compare():
    assert 10==10

returns True.

We can also write simple assertions like,

assert True, a successful assertion.
assert False, an assertion failure.

Also, note that if an assertion failure occurs, it stops the execution of that test at that point and goes to next test.

Basic Test:

Let’s start with a basic test now,

Create any directory like, PyTestProject\Arithmetic and create the following file under it.

arithmetic_test.py

def test_add():
    a=6
    b=3
    assert a + b == 9, “Expected: {}, Result: {}”.format(a+b,9)
 
 def testsubtract():
     a=6
     b=3
     assert a-b == 2, “Expected:{}, Result: {}”.format(a-b,2)
 
 def multiplytest():
     a=6
     b=3
     assert a * b == 6, “Expected: {}, Result: {}”.format(a*b,6)
 
 def divide():
     a=6
     b=3
     assert a/b == 2, “Expected:{}, Result: {}”.format(a/b,2)

Understanding the output:

C:\PycharmProjects\PyTestProject\Arithmetic>pytest
=======test session starts ========================================
platform win32 — Python 3.7.7, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: C:\PycharmProjects\PyTestProject\Arithmetic
collected 2 items
arithmetic_test.py .F [100%]
=====================================================================================FAILURES =======================================
___________________________________________________________________                  testsubtract ___________________________________________________________________
def testsubtract():
a=6
b=3
> assert a-b == 2, “Expected:{}, Result: {}”.format(a-b,2)
E AssertionError: Expected:3, Result: 2
E assert (6–3) == 2
arithmetic_test.py:9: AssertionError
===================short test summary info ========================================
FAILED arithmetic_test.py::testsubtract — AssertionError: Expected:3, Result: 2
===========================1 failed, 1 passed in 0.21s ========================================

First, it shows the environment we are running on, Python and PyTest versions and any plug-ins installed. Second, it shows the root directory where the tests reside. Third, it shows the tests identified. Here 2 as multiplytest() and divide() are not prefixed with “test” keyword. Fourth, it shows the filename and result of tests in that file (.) dot indicates test Pass F indicates test Fail E indicates an exception occurred Fifth, it shows the detailed report of failure tests, as we included detailed assertions the report is more clear with expected and result values shown explicitly. Sixth, again shows a short test summary info for respective tests.

Also, note that even, if the first test failed, it went to the next text.

Try the following code to pass all the tests and understand the output.

arithmetic_test.py

def test_add():
    a=6
    b=3
    assert a + b == 9, “Expected: {}, Result: {}”.format(a+b,9)
 
def testsubtract():
    a=6
    b=3
    assert a-b == 3, “Expected:{}, Result: {}”.format(a-b,3)
 
def multiplytest():
    a=6
    b=3
    assert a * b == 18, “Expected: {}, Result: {}”.format(a*b,18)
 
def divide():
    a=6
    b=3
    assert a/b == 2, “Expected:{}, Result: {}”.format(a/b,2)

Output:

C:\PycharmProjects\PyTestProject\Arithmetic>pytest
===================================================================================== test session starts ====================================================================
platform win32 — Python 3.7.7, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: C:\PycharmProjects\PyTestProject\Arithmetic
collected 2 items
arithmetic_test.py .. [100%]
===================================================================================== 2 passed in 0.25s ====================================================================

Here, collected 2 tests and 2 dots after filename indicate that 2 tests have passed.

Fixtures

If you have observed the above example, the numbers are initialized in every test separately. What if, we can initialize once and used them in all tests. That’s what the fixtures are used for. They help us to assign initial values or configure a setup like connecting to a database.

A fixture is also a definition but marked as a fixture using: @pytest.fixture

Let’s, directly dive into an example:

import pytest@pytest.fixture

def numbers():
    a=6
    b=3
    return a,b
    
def test_add(numbers):
    assert numbers[0] + numbers[1] == 5, \
            "Expected: {}, Result {}".format(numbers[0]+numbers[1],5)
            
def testsubtract(numbers):
    assert numbers[0] - numbers[1] == 3, \
            "Expected:{}, Result:{}".format(numbers[0] - numbers[1], 3)
            
def testmultiply(numbers):    
    assert numbers[0] * numbers[1] == 18, \
           "Expected:{}, Result:{}".format(numbers[0] * numbers[1], 18)
            
def test_divide(numbers):
    assert (numbers[0] / numbers[1]) == 3, \        
            "Expected:{}, Result:{}".format(numbers[0] / numbers[1], 3)
            

Here, we need to import pytest module to make the fixture identified.

Now the definition numbers() is marked as fixture. It is used to reduce repetition of code. Here the fixture is used to initialize the numbers.

@pytest.fixture
def numbers():
    a=6
    b=3
    return a,b

The tests that need to use the fixture, must send the name of the fixture as an input parameter to the test. This way, the fixture is attached to the test. Before the test, the mentioned fixture is executed and returns the value/s to the test which can be accessed by the test through the input parameter. Here it is “numbers”.

def test_add(numbers):
    assert numbers[0] + numbers[1] == 5, \
             “Expected: {}, Result: {}”.format(numbers[0]+numbers[1],5)

Now the output, -v used to increase the verbosity.

C:\Users\PyTestProject>pytest arithmetic_test.py -v
===================================================================================== test session starts ====================================================================
platform win32 — Python 3.7.7, pytest-5.4.3, py-1.8.2, pluggy-0.13.1 — c:\programs\python\python37\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\PyTestProject
collected 5 items / 1 deselected / 4 selected
arithmetic_test.py::test_add FAILED [ 25%]
arithmetic_test.py::testsubtract PASSED [ 50%]
arithmetic_test.py::testmultiply PASSED [ 75%]
arithmetic_test.py::test_divide FAILED [100%]
===================================================================================== FAILURES ====================================================================
____________________________________________________________________________________test_add ____________________________________________________________________
numbers = (6, 3)def test_add(numbers):
> assert numbers[0] + numbers[1] == 5, \“Expected: {}, Result: {}”.format(numbers[0]+numbers[1],5)
E AssertionError: Expected: 9, Result: 5
E assert 9 == 5E +9E -5
arithmetic_test.py:10: AssertionError
_____________________________________________________________________________________ test_divide ____________________________________________________________________
numbers = (6, 3)def test_divide(numbers):
> assert (numbers[0] / numbers[1]) == 3, \“Expected: {}, Result: {}”.format(numbers[0] / numbers[1], 3)
E AssertionError: Expected: 2.0, Result: 3
E assert 2.0 == 3E +2.0E -3
arithmetic_test.py:22: AssertionError
=====================================================================================short test summary info ====================================================================
FAILED arithmetic_test.py::test_add — AssertionError: Expected: 9, Result: 5
FAILED arithmetic_test.py::test_divide — AssertionError: Expected: 2.0, Result: 3
==================================================================================== 2 failed, 2 passed, 1 deselected in 0.27s ====================================================================

To view, all the available fixtures (custom and built-in) use the following command:

pytest --fixtures

conftest.py

The drawback of fixtures is that, they cannot be accessible outside that particular file, hence Pytest provided us with conftest.py.

conftest.py is a file created in the same directory where the tests are located. It defines the fixtures, that are needed for the tests. Therefore, it reduces duplicity of code as multiple test files can access the same fixture/s.

Now an example,

Create a file conftest.py in the same directory and add this code.

import pytest
 
 @pytest.fixture
 def numbers():
     a=6
     b=3
     return a,b

Modify the arithmetic_test.py by removing the fixture.

def test_add(numbers):
     assert numbers[0] + numbers[1] == 5, \
           “Expected: {}, Result: {}”.format(numbers[0]+numbers[1],5)
 
 def testsubtract(numbers):
     assert numbers[0] — numbers[1] == 3, \
          “Expected: {}, Result: {}”.format(numbers[0] — numbers[1], 3)
 
 def testmultiply(numbers):
     assert numbers[0] * numbers[1] == 18, \
         “Expected: {}, Result: {}”.format(numbers[0] * numbers[1], 18)
 
 def test_divide(numbers):
     assert (numbers[0] / numbers[1]) == 3, \
          “Expected: {}, Result: {}”.format(numbers[0] / numbers[1], 3)

Add another test file test_compare.py that uses the same fixture in conftest.py.

def test_small(numbers):
    assert numbers[0] < numbers[1],\
         “{} not less than {}”.format(numbers[0],numbers[1])
 
def test_greater(numbers):
    assert numbers[0] > numbers[1],\
          “{} not greater than {}”.format(numbers[0],numbers[1])
  
def test_equalto(numbers):
    assert numbers[0] == numbers[1],\
         “{} not equal to {}”.format(numbers[0],numbers[1])

Let’s run and check the output,

When you run this, arithmetic_test.py and test_compare.py run using the fixture in conftest.py. Any other files in this directory will not be executed if they are if not prefixed by test_ or suffixed by _test. Total tests collected here are 7.

C:\Users\PyTestProject>pytest
=====================================================================================test session starts ====================================================================
platform win32 — Python 3.7.7, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: C:\Users\PyTestProject
collected 7 items
arithmetic_test.py F..F [ 57%]
test_compare.py F.F [100%]
=====================================================================================FAILURES ====================================================================
_____________________________________________________________________________________test_add ____________________________________________________________________
numbers = (6, 3)
def test_add(numbers):
> assert numbers[0] + numbers[1] == 5, \“Expected: {}, Result: {}”.format(numbers[0]+numbers[1],5)
E AssertionError: Expected: 9, Result: 5
E assert (6 + 3) == 5
arithmetic_test.py:3: AssertionError
_____________________________________________________________________________________test_divide ____________________________________________________________________
numbers = (6, 3)
def test_divide(numbers):
> assert (numbers[0] / numbers[1]) == 3, \“Expected: {}, Result: {}”.format(numbers[0] / numbers[1], 3)
E AssertionError: Expected: 2.0, Result: 3
E assert (6 / 3) == 3
arithmetic_test.py:15: AssertionError
_____________________________________________________________________________________test_small ____________________________________________________________________
numbers = (6, 3)
def test_small(numbers):
> assert numbers[0] < numbers[1],\“{} not less than {}”.format(numbers[0],numbers[1])
E AssertionError: 6 not less than 3
E assert 6 < 3
test_compare.py:2: AssertionError
_____________________________________________________________________________________test_equalto ____________________________________________________________________
numbers = (6, 3)
def test_equalto(numbers):
> assert numbers[0] == numbers[1],\“{} not equal to {}”.format(numbers[0],numbers[1])
E AssertionError: 6 not equal to 3
E assert 6 == 3test_compare.py:11: AssertionError
=====================================================================================short test summary info ====================================================================
FAILED arithmetic_test.py::test_add — AssertionError: Expected: 9, Result: 5
FAILED arithmetic_test.py::test_divide — AssertionError: Expected: 2.0, Result: 3
FAILED test_compare.py::test_small — AssertionError: 6 not less than 3
FAILED test_compare.py::test_equalto — AssertionError: 6 not equal to 3
=====================================================================================4 failed, 3 passed in 3.23s ====================================================================

Here, both the test files used the fixture in conftest.py to initialize values and then execute the test.

Parallel execution

Now that we are beginning with PyTest , we have simple tests that run in seconds. But, if we consider a real time scenario, there will be multiple test suites with multiple test files and test definitions, which would take several hours to execute. Hence, parallel execution makes PyTest even more powerful.

For this, install the plug-in pytest-xdist as,

pip install pytest-xdist

With this plug-in ,we can assign multiple workers to run the tests in parallel by running the command,

pytest -n 4

-n assigns multiple workers, ex: 4.

Below we are running with 2 workers on test files arithmetic_test.py and test_compare.py and examine the output. Though there is no much time difference found here, we can see 2 workers assigned gw0 and gw1, which are highlighted as bold for better understanding.

C:\Users\PyTestProject>pytest -n 2
=====================================================================================test session starts ====================================================================
platform win32 — Python 3.7.7, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: C:\Users\PyTestProject
plugins: forked-1.1.3, xdist-1.32.0
gw0 [7] / gw1 [7]
.FFF..F [100%]
===================================================================================== FAILURES ====================================================================
_____________________________________________________________________________________ test_divide ____________________________________________________________________

[gw1] win32 — Python 3.7.7 c:\users\prade\appdata\local\programs\python\python37\python.exe
numbers = (6, 3)
def test_divide(numbers):
> assert (numbers[0] / numbers[1]) == 3, \“Expected: {}, Result: {}”.format(numbers[0] / numbers[1], 3)
E AssertionError: Expected: 2.0, Result: 3
E assert (6 / 3) == 3
arithmetic_test.py:15: AssertionError
_____________________________________________________________________________________test_add ____________________________________________________________________

[gw0] win32 — Python 3.7.7 c:\users\prade\appdata\local\programs\python\python37\python.exe
numbers = (6, 3)
def test_add(numbers):
> assert numbers[0] + numbers[1] == 5, \“Expected: {}, Result: {}”.format(numbers[0]+numbers[1],5)
E AssertionError: Expected: 9, Result: 5
E assert (6 + 3) == 5
arithmetic_test.py:3: AssertionError
_____________________________________________________________________________________ test_small ____________________________________________________________________

[gw1] win32 — Python 3.7.7 c:\users\prade\appdata\local\programs\python\python37\python.exe
numbers = (6, 3)
def test_small(numbers)
:> assert numbers[0] < numbers[1],\“{} not less than {}”.format(numbers[0],numbers[1])
E AssertionError: 6 not less than 3
E assert 6 < 3
test_compare.py:2: AssertionError
_____________________________________________________________________________________ test_equalto ____________________________________________________________________

[gw0] win32 — Python 3.7.7 c:\users\prade\appdata\local\programs\python\python37\python.exe
numbers = (6, 3)
def test_equalto(numbers):
> assert numbers[0] == numbers[1],\“{} not equal to {}”.format(numbers[0],numbers[1])
E AssertionError: 6 not equal to 3
E assert 6 == 3
test_compare.py:11: AssertionError
=====================================================================================short test summary info ====================================================================
FAILED arithmetic_test.py::test_divide — AssertionError: Expected: 2.0, Result: 3
FAILED arithmetic_test.py::test_add — AssertionError: Expected: 9, Result: 5
FAILED test_compare.py::test_small — AssertionError: 6 not less than 3
FAILED test_compare.py::test_equalto — AssertionError: 6 not equal to 3
===================================================================================== 4 failed, 3 passed in 3.16s ====================================================================

If you have closely observed, the 2 workers have run in parallel to run different tests.

In real time, if we are running selenium tests then multiple browsers can be opened at once and different tests can be performed parallelly which helps in quick testing and turn around.

Conclusion

Hope, this gives you a good head start to have your hands on the PyTest framework. Here, we have gone through the installation to creating a basic test with assertions, adding fixtures and conftest.py and a quick look on parallel testing. The power of PyTest, also comes from grouping of tests to be executed or skipped through “Markers”, which will discussed in my future blog/s.

You can also access the code given here, on my GitHub.

Source