pycon siberia 2016. Не доверяйте тестам!
TRANSCRIPT
![Page 1: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/1.jpg)
Цыганов Иван Positive Technologies
Не доверяйте тестам!
![Page 2: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/2.jpg)
Обо мне
✤ Спикер PyCon Russia 2016, PiterPy#2 и PiterPy#3
✤ Люблю OpenSource
✤ Не умею frontend
![Page 3: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/3.jpg)
✤ 15 лет практического опыта на рынке ИБ
✤ Более 650 сотрудников в 9 странах
✤ Каждый год находим более 200 уязвимостей нулевого дня
✤ Проводим более 200 аудитов безопасности в крупнейших компаниях мира ежегодно
![Page 4: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/4.jpg)
![Page 5: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/5.jpg)
MaxPatrol
✤ Тестирование на проникновение (Pentest)
✤ Системные проверки (Audit)
✤ Соответствие стандартам (Compliance)
✤ Одна из крупнейших баз знаний в мире
Система контроля защищенности и соответствия стандартам.
![Page 6: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/6.jpg)
✤ Тестирование на проникновение (Pentest)
✤ Системные проверки (Audit)
✤ Соответствие стандартам (Compliance)
✤ Одна из крупнейших баз знаний в мире
Система контроля защищенности и соответствия стандартам.
✤Системные проверки (Audit)
MaxPatrol
![Page 7: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/7.jpg)
> 50 000 строк кода
![Page 8: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/8.jpg)
Зачем тестировать?
✤ Уверенность, что написанный код работает
✤ Ревью кода становится проще
✤ Гарантия, что ничего не сломалось при изменениях
![Page 9: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/9.jpg)
есть тесты != код протестирован
![Page 10: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/10.jpg)
Давайте писать тесты!
def get_total_price(cart_prices): if len(cart_prices) == 0: return result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return result['TotalPrice'] - result.get('Discount')
![Page 11: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/11.jpg)
Плохой тест
def get_total_price(cart_prices): if len(cart_prices) == 0: return result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return result['TotalPrice'] - result.get('Discount')
def test_get_total_price(): assert get_total_price([90, 10]) == 75
![Page 12: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/12.jpg)
Неожиданные данные
>>> balance = 1000 >>> >>> goods = [] >>> >>> balance -= get_total_price(goods)
Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for -=: 'int' and 'NoneType' >>>
![Page 13: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/13.jpg)
есть тесты == есть тесты
![Page 14: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/14.jpg)
Как сделать тесты лучше?
✤ Проверить покрытие кода тестами
✤ Попробовать мутационное тестирование
![Page 15: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/15.jpg)
coverage.py
✤ Позволяет проверить покрытие кода тестами
✤ Есть плагин для pytest
![Page 16: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/16.jpg)
coverage.py
✤ Позволяет проверить покрытие кода тестами
✤ Есть плагин для pytest
✤ В основном работает
![Page 17: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/17.jpg)
coverage.ini
[report]show_missing = Trueprecision = 2
py.test --cov-config=coverage.ini --cov=target test.py
![Page 18: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/18.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
def test_get_total_price(): assert get_total_price([90, 10]) == 75
Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 1 85.71% 2
![Page 19: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/19.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
def test_get_total_price(): assert get_total_price([90, 10]) == 75
Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 1 85.71% 2
2 if len(cart_prices) == 0:
![Page 20: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/20.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0
Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 0 100.00%
![Page 21: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/21.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0
Name Stmts Miss Cover Missing -------------------------------------------- target.py 7 0 100.00%
![Page 22: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/22.jpg)
>>> get_total_price([90])
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
![Page 23: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/23.jpg)
>>> get_total_price([90]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 9, in get_total_price TypeError: unsupported operand type(s) for -: 'int' and 'NoneType' >>>
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get('Discount')
![Page 24: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/24.jpg)
![Page 25: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/25.jpg)
coverage.ini
[report]show_missing = Trueprecision = 2[run]branch = True
py.test --cov-config=coverage.ini --cov=target test.py
![Page 26: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/26.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 1 90.91% 6 ->9
![Page 27: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/27.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 1 90.91% 6 ->9
![Page 28: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/28.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 7 0 4 0 100.00%
![Page 29: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/29.jpg)
![Page 30: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/30.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
![Page 31: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/31.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price)
def test_get_total_price(): assert get_total_price([90, 10]) == 75
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 1 4 1 80.00% 3, 2 ->3
![Page 32: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/32.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 0 4 0 100.00%
![Page 33: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/33.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 total_price = sum(cart_prices) 6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25 7 8 return total_price-get_discount(cart_prices, total_price)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 6 0 4 0 100.00%
6 get_discount = lambda items, price: len(items) >= 2 and price * 0.25
![Page 34: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/34.jpg)
![Page 35: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/35.jpg)
Как считать coverage?
Все строкиРеально выполненные
строки- Непокрытые строки=
![Page 36: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/36.jpg)
Все строки
Source
coverage.parser.PythonParser
Statements
![Page 37: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/37.jpg)
coverage.parser.PythonParser
✤ Обходит все токены и отмечает «интересные» факты
✤ Компилирует код. Обходит code-object и сохраняет номера строк
![Page 38: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/38.jpg)
Обход токенов
✤ Запоминает определения классов
✤ «Сворачивает» многострочные выражения
✤ Исключает комментарии
![Page 39: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/39.jpg)
Обход байткода
✤ Полностью повторяет метод dis.findlinestarts
✤ Анализирует code_obj.co_lnotab
✤ Генерирует пару (номер байткода, номер строки)
![Page 40: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/40.jpg)
Как считать coverage --branch?
Все переходыРеально выполненные
переходы- Непокрытые переходы=
![Page 41: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/41.jpg)
Все переходы
Source
coverage.parser.AstArcAnalyzer
(from_line, to_line)
coverage.parser.PythonParser
![Page 42: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/42.jpg)
coverage.parser.AstArcAnalyzer
✤ Обходит AST с корневой ноды
✤ Обрабатывает отдельно каждый тип нод отдельно
![Page 43: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/43.jpg)
Обработка ноды
class While(stmt): _fields = ( 'test', 'body', 'orelse', )
while i<10: print(i) i += 1
![Page 44: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/44.jpg)
Обработка ноды
class While(stmt): _fields = ( 'test', 'body', 'orelse', )
while i<10: print(i) i += 1 else: print('All done')
![Page 45: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/45.jpg)
Выполненные строки
sys.settrace(tracefunc)Set the system’s trace function, which allows you to implement a Python source code debugger in Python.
Trace functions should have three arguments: frame, event, and arg. frame is the current stack frame. event is a string: 'call', 'line', 'return', 'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the event type.
![Page 46: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/46.jpg)
PyTracer «call» event
✤ Сохраняем данные предыдущего контекста
✤ Начинаем собирать данные нового контекста
✤ Учитываем особенности генераторов
![Page 47: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/47.jpg)
PyTracer «line» event
✤ Запоминаем выполняемую строку
✤ Запоминаем переход между строками
![Page 48: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/48.jpg)
PyTracer «return» event
✤ Отмечаем выход из контекста
✤ Помним о том, что yield это тоже return
![Page 49: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/49.jpg)
Отчет
✤ Что выполнялось
✤ Что должно было выполниться
✤ Ругаемся
![Page 50: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/50.jpg)
Зачем такие сложности?
1 for i in some_list: 2 if i == 'Hello': 3 print(i + ' World!') 4 elif i == 'Skip': 5 continue 6 else: 7 break 8 else: 9 print(r'¯\_(ツ)_/¯')
![Page 51: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/51.jpg)
Серебряная пуля?
![Page 52: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/52.jpg)
Не совсем…
![Page 53: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/53.jpg)
Что может пойти не так?
1 def make_dict(a,b,c): 2 return { 3 'a': a, 4 'b': b if a>1 else 0, 5 'c': [ 6 i for i in range(c) if i<(a*10) 7 ] 6 }
![Page 54: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/54.jpg)
![Page 55: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/55.jpg)
Мутационное тестирование
✤ Берем тестируемый код
✤ Мутируем
✤ Тестируем мутантов нашими тестами
✤ Тест не упал -> плохой тест
![Page 56: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/56.jpg)
Мутационное тестирование
✤ Берем тестируемый код
✤ Мутируем
✤ Тестируем мутантов нашими тестами
✤ Если тест не упал -> это плохой тест✤ Тест не упал -> плохой тест
![Page 57: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/57.jpg)
Идея
def mul(a, b): return a * b
def test_mul(): assert mul(2, 2) == 4
![Page 58: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/58.jpg)
Идея
def mul(a, b): return a * b
def test_mul(): assert mul(2, 2) == 4
def mul(a, b): return a ** b
![Page 59: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/59.jpg)
Идея
def mul(a, b): return a * b
def test_mul(): assert mul(2, 2) == 4
def mul(a, b): return a + b
def mul(a, b): return a ** b
![Page 60: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/60.jpg)
Идея
def mul(a, b): return a * b
def test_mul(): assert mul(2, 2) == 4 assert mul(2, 3) == 6
def mul(a, b): return a + b
def mul(a, b): return a ** b
![Page 61: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/61.jpg)
Tools
MutPy
✤ Проект заброшен
cosmic-ray
✤ Активно развивается
✤ Требует RabbitMQ
![Page 62: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/62.jpg)
Реализация
Source
NodeTransformer
compile
run test
![Page 63: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/63.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] / 0.25 8 …
![Page 64: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/64.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 9 return result['TotalPrice'] + result.get(‘Discount’, 0) …
![Page 65: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/65.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 2 if (not len(cart_prices) == 0): 3 return 0 …
![Page 66: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/66.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 2 if len(cart_prices) == 1: 3 return 0 …
![Page 67: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/67.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 2 if len(cart_prices) == 0: 3 return 1 …
![Page 68: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/68.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 5 result = {'': sum(cart_prices)} …
![Page 69: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/69.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
… 9 return result[‘some_key'] - result.get(‘Discount’, 0)
![Page 70: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/70.jpg)
Мутации
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
![Page 71: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/71.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90)
[*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
![Page 72: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/72.jpg)
1 def get_total_price(cart_prices): 2 if len(cart_prices) == 0: 3 return 0 4 5 result = {'TotalPrice': sum(cart_prices)} 6 if len(cart_prices) >= 2: 7 result['Discount'] = result['TotalPrice'] * 0.25 8 9 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90)
[*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%)
- incompetent: 0 (0.0%) - timeout: 0 (0.0%)
- survived: 1 (3.6%)
![Page 73: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/73.jpg)
… ---------------------------------------------------------- 1: def get_total_price(cart_prices): 2: if len(cart_prices) == 0: ~3: pass 4: 5: result = {'TotalPrice': sum(cart_prices)} 6: if len(cart_prices) >= 2: 7: result['Discount'] = result['TotalPrice'] * 0.25 8: ---------------------------------------------------------- [0.00968 s] survived - [# 26] SDL target:5 :
…
[*] Mutation score [0.50795 s]: 96.4% - all: 28 - killed: 27 (96.4%) - survived: 1 (3.6%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
![Page 74: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/74.jpg)
1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(self): self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90)
[*] Mutation score [0.44658 s]: 100.0% - all: 23 - killed: 23 (100.0%) - survived: 0 (0.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
![Page 75: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/75.jpg)
Идея имеет право на жизнь и работает!
Но требует много ресурсов.
![Page 76: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/76.jpg)
1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90
Name Stmts Miss Cover Missing -------------------------------------------- target.py 5 0 100.00%
![Page 77: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/77.jpg)
1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 5 0 2 0 100.00%
![Page 78: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/78.jpg)
1 def get_total_price(cart_prices): 2 result = {'TotalPrice': sum(cart_prices)} 3 if len(cart_prices) >= 2: 4 result['Discount'] = result['TotalPrice'] * 0.25 5 6 return result['TotalPrice'] - result.get(‘Discount’, 0)
def test_get_total_price(): assert get_total_price([90, 10]) == 75 assert get_total_price( []) == 0 assert get_total_price([90]) == 90
Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------- target.py 5 0 2 0 100.00%
![Page 79: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/79.jpg)
Есть тесты != код протестирован
![Page 80: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/80.jpg)
Есть тесты != код протестирован
Качество тестов важнее количества
![Page 81: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/81.jpg)
Есть тесты != код протестирован
Качество тестов важнее количества
100% coverage - не повод расслабляться
![Page 82: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/82.jpg)
![Page 83: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/83.jpg)
![Page 84: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/84.jpg)
Simple app
app = Flask(__name__) @app.route('/get_total_discount', methods=['POST']) def get_total_discount(): cart_prices = json.loads(request.form['cart_prices']) result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return jsonify(result['TotalPrice'] - result.get('Discount', 0))
flask_app.py
![Page 85: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/85.jpg)
pip install pytest-flask
@pytest.fixture def app(): from flask_app import app return app def test_get_total_discount(client): get_total_discount = lambda prices: client.post( '/get_total_discount', data=dict(cart_prices=json.dumps(prices)) ).json assert get_total_discount([90, 10]) == 75 assert get_total_discount( []) == 0 assert get_total_discount([90]) == 90
test_flask_app.py
![Page 86: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/86.jpg)
pip install pytest-flask
Name Stmts Miss Cover Missing ----------------------------------------------- flask_app.py 9 0 100.00%
py.test --cov-config=coverage.ini \ --cov=flask_app \ test_flask_app.py
Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------- flask_app.py 9 0 2 0 100.00%
py.test --cov-config=coverage_branch.ini \ --cov=flask_app \ test_flask_app.py
![Page 87: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/87.jpg)
mutpy
class FlaskTestCase(unittest.TestCase): def setUp(self): self.app = flask_app.app.test_client() def post(self, path, data): return json.loads(self.app.post(path, data=data).data.decode('utf-8')) def test_get_total_discount(self): get_total_discount = lambda prices: self.post( '/get_total_discount', data=dict(cart_prices=json.dumps(prices)) ) self.assertEqual(get_total_discount([90, 10]), 75)
unittest_flask_app.py
![Page 88: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/88.jpg)
mutpy
[*] Mutation score [0.39122 s]: 100.0% - all: 27 - killed: 1 (3.7%) - survived: 0 (0.0%) - incompetent: 26 (96.3%) - timeout: 0 (0.0%)
mut.py --target flask_app --unit-test unittest_flask_app
![Page 89: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/89.jpg)
mutpy
[*] Mutation score [0.39122 s]: 100.0% - all: 27 - killed: 1 (3.7%) - survived: 0 (0.0%)
- incompetent: 26 (96.3%) - timeout: 0 (0.0%)
mut.py --target flask_app --unit-test unittest_flask_app
![Page 90: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/90.jpg)
mutpy
def _matching_loader_thinks_module_is_package(loader, mod_name): #... raise AttributeError( ('%s.is_package() method is missing but is required by Flask of ' 'PEP 302 import hooks. If you do not use import hooks and ' 'you encounter this error please file a bug against Flask.') % loader.__class__.__name__)
![Page 91: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/91.jpg)
mutpy
def _matching_loader_thinks_module_is_package(loader, mod_name): #... raise AttributeError( ('%s.is_package() method is missing but is required by Flask of ' 'PEP 302 import hooks. If you do not use import hooks and ' 'you encounter this error please file a bug against Flask.') % loader.__class__.__name__)
class InjectImporter: def __init__(self, module): # ... def find_module(self, fullname, path=None): # ... def load_module(self, fullname): # ... def install(self): # ... def uninstall(cls): # ...
![Page 92: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/92.jpg)
mutpy
class InjectImporter: def __init__(self, module): # ... def find_module(self, fullname, path=None): # ... def load_module(self, fullname): # ... def install(self): # ... def uninstall(cls): # … def is_package(self, fullname): # ...
![Page 93: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/93.jpg)
mutpy
[*] Mutation score [1.14206 s]: 100.0% - all: 27 - killed: 25 (92.6%) - survived: 0 (0.0%) - incompetent: 2 (7.4%) - timeout: 0 (0.0%)
mut.py --target flask_app --unit-test unittest_flask_app
![Page 94: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/94.jpg)
![Page 95: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/95.jpg)
Simple app
import jsonfrom django.http import HttpResponse def index(request): cart_prices = json.loads(request.POST['cart_prices']) result = {'TotalPrice': sum(cart_prices)} if len(cart_prices) >= 2: result['Discount'] = result['TotalPrice'] * 0.25 return HttpResponse(result['TotalPrice'] - result.get('Discount', 0))
django_root/billing/views.py
![Page 96: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/96.jpg)
pip install pytest-django
class TestCase1(TestCase): def test_get_total_price(self): get_total_price = lambda items: json.loads( self.client.post( '/billing/', data={'cart_prices': json.dumps(items)} ).content.decode('utf-8') ) self.assertEqual(get_total_price([90, 10]), 75) self.assertEqual(get_total_price( []), 0) self.assertEqual(get_total_price([90]), 90)
django_root/billing/tests.py
![Page 97: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/97.jpg)
pip install pytest-django
Name Stmts Miss Cover Missing --------------------------------------------------- billing/views.py 8 0 100.00%
py.test --cov-config=coverage.ini \ --cov=billing.views \ billing/tests.py
Name Stmts Miss Branch BrPart Cover Missing ----------------------------------------------------------------- billing/views.py 8 0 2 0 100.00%
py.test --cov-config=coverage_branch.ini \ --cov=billing.views \ billing/tests.py
![Page 98: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/98.jpg)
mutpy
[*] Start mutation process: - targets: billing.views - tests: billing.tests [*] Tests failed: - error in setUpClass (billing.tests.TestCase1) - django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
mut.py --target billing.views --unit-test billing.tests
![Page 99: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/99.jpg)
mutpy
class Command(BaseCommand): def handle(self, *args, **options): operators_set = operators.standard_operators if options['experimental_operators']: operators_set |= operators.experimental_operators controller = MutationController( target_loader=ModulesLoader(options['target'], None), test_loader=ModulesLoader(options['unit_test'], None), views=[TextView(colored_output=False, show_mutants=True)], mutant_generator=FirstOrderMutator(operators_set) ) controller.run()
django_root/mutate_command/management/commands/mutate.py
![Page 100: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/100.jpg)
mutpy
[*] Mutation score [1.07321 s]: 0.0% - all: 22 - killed: 0 (0.0%) - survived: 22 (100.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
python manage.py mutate \ --target billing.views --unit-test billing.tests
![Page 101: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/101.jpg)
mutpy
class RegexURLPattern(LocaleRegexProvider): def __init__(self, regex, callback, default_args=None, name=None): LocaleRegexProvider.__init__(self, regex) self.callback = callback # the view self.default_args = default_args or {} self.name = name
django.urls.resolvers.RegexURLPattern
![Page 102: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/102.jpg)
mutpyimport importlib
class Command(BaseCommand): def hack_django_for_mutate(self): def set_cb(self, value): self._cb = value def get_cb(self): module = importlib.import_module(self._cb.__module__) return module.__dict__.get(self._cb.__name__)
import django.urls.resolvers as r r.RegexURLPattern.callback = property(callback, set_cb) def __init__(self, *args, **kwargs): self.hack_django_for_mutate() super().__init__(*args, **kwargs) def add_arguments(self, parser): # ...
![Page 103: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/103.jpg)
mutpy
[*] Mutation score [1.48715 s]: 100.0% - all: 22 - killed: 22 (100.0%) - survived: 0 (0.0%) - incompetent: 0 (0.0%) - timeout: 0 (0.0%)
python manage.py mutate \ --target billing.views --unit-test billing.tests
![Page 105: PyCon Siberia 2016. Не доверяйте тестам!](https://reader037.vdocuments.pub/reader037/viewer/2022102715/588327961a28abe2758b6939/html5/thumbnails/105.jpg)
Links
✤ https://github.com/pytest-dev/pytest
✤ https://github.com/pytest-dev/pytest-flask
✤ https://github.com/pytest-dev/pytest-django
✤ https://bitbucket.org/ned/coveragepy
✤ https://github.com/pytest-dev/pytest-cov
✤ https://bitbucket.org/khalas/mutpy
✤ https://github.com/sixty-north/cosmic-ray