Characterization Testing for Legacy Applications

# Characterization Testing for Legacy Applications
## Day Camp 4 Developers, 18 Sep 2015
## Paul M. Jones
### <https://joind.in/15189>

---

# Read These

![Refactoring](./images/refactoring.jpg)
![Working Effectively with Legacy Code](./images/welc.jpg)

---

# About Me

- 8 years USAF Intelligence
- BASIC in 1983, PHP since 1999
- Jr. Developer, VP Engineering
- Aura project, Zend_DB, Zend_View
- PHP-FIG, PSR-1, PSR-2, PSR-4
- [Action-Domain-Responder](http://pmjones.io/adr)
- [MLAPHP](http://mlaphp.com)

---

# Debugging and Tests

- Fixing one bug might cause another
- Tests help you keep code debugged
- Legacy code probably doesn't have tests
- Hard to tell if legacy bug *stays fixed*

---

# What Does "Legacy" Mean? (1/2)

- Page scripts directly in the document root
- Special index files to prevent directory access
- Special logic at the top of some files to `die()` or `exit()`
- Include-oriented instead of class-oriented or object-oriented

---

# What Does "Legacy" Mean? (2/2)

- Relatively few classes
- Poor class structure
- Relies more heavily on functions than on class methods
- Page scripts, classes, and functions combine MVC concerns
- One or more incomplete attempts at a rewrite
- No automated test suite

---

# Legacy Code Example

[legacy.php](./legacy.php)

---

# A Shorter Definition of "Legacy"

"To me, *legacy code* is simply code without tests."

*-- Michael Feathers*

---

# How To Debug Safely Without Tests?

---

# Characterization Tests

---

# What is Characterization Testing?

- (Unit | integration | system | functional) tests what it *ought to do*
- Characterization tests what it *actually does* ...
- ... and that it *does not change* after code modifications
- Flip side of acceptance testing (is *vs* ought)
- Black box: no knowledge of codebase internals

---

# Regression Management

- Unit test fails?
    - Usually fix the code.
    - Rarely fix the test.

- Characterization test fails?
    - Unintended change? Fix the code.
    - Intended change? Fix the test!

---

# Characterization Tracks Existing Output

- These tests are updated to track what the system *actually does*
- As what it *actually does* changes intentionally, update the tests

---

# When To Use Characterization Testing

- Only when there are no units to test
- If you *have* units, test those first
- Characterization is a supplement, not a replacement

---

# The Way Of Testivus

- "More testing karma, less testing dogma."
- "An imperfect test today is better than a perfect test someday."
- "Write the best test you can, *today.*"

---

# Getting Started With Characterizaton Testing

---

# Prerequisites

- **Do not** test against production
- Set up a virtual environment
- Vagrant is great for this purpose

---

# Tool: Codeception

- http://codeception.com
- Black box testing
- Emulates a browser

```php
$I->amOnPage('/login');
$I->fillField(['name' => 'username'], 'kotter');
$I->fillField(['name' => 'password'], 'horshack');
$I->click('Login');
$I->see('Welcome Back Kotter!');
```

---

# Installing Codeception

```bash
wget http://codeception.com/codecept.phar .
chmod +x codecept.phar
```

---

# Tool: PhantomJS

- http://phantomjs.org
- Headless WebKit scriptable (JS, CSS, DOM, SVG, Canvas)

```bash
sudo apt-get install phantomjs
phantomjs --webdriver=4444
```

---

# Bootstrapping Codeception

- In development environment, at repository root:

```bash
./codecept.phar bootstrap
```

---

# Codeception Files

```bash
├── codecept.phar
├── codeception.yml
└── tests
    ├── _bootstrap.php
    ├── _data
    ├── _envs
    ├── _output
    ├── _support
    ├── acceptance
    ├── acceptance.suite.yml
    ├── functional
    ├── functional.suite.yml
    ├── unit
    └── unit.suite.yml
```

---

# Create "Character" Suite

* Co-opt "acceptance" for "character"

```bash
cd tests
cp -r acceptance character
cp acceptance.suite.yml character.suite.yml
```

```bash
└── tests
    ├── ...
    ├── character
    ├── character.suite.yml
    ├── ...
```

---

# Edit character.suite.yml

```
class_name: AcceptanceTester
modules:
    enabled:
        - WebDriver:
            browser: firefox
            url: http://example.dev/
        - \Helper\Acceptance
```

---

# Our First Characterization Test

* `tests/character/HomeCept.php`

```php
$I = new AcceptanceTester($scenario);
$I->wantTo('See the home page');
$I->amOnPage('/');
$I->see('My Home Page Title');
$I->dontSee('Fatal error:');
$I->dontSee('Warning:');
$I->dontSee('Notice:');
```

---

# Running The Test

```
$ ./codecept.phar run character
Codeception PHP Testing Framework v2.1.2
Powered by PHPUnit 4.8.2 by Sebastian Bergmann and contributors.

Character Tests (1) ----------------------------------------------------
See the home page (HomeCept)                                        Ok
------------------------------------------------------------------------

Time: 1.8 seconds, Memory: 10.25Mb

OK (1 test, 4 assertions)
$
```

---

# Break Something

* In your application code on the home page:

```php
error_reporting(E_ALL);
ini_set('display_errors', true);
trigger_error("Broken!", E_USER_ERROR);
```

---

```
$ ./codecept.phar run character
...
There was 1 failure:

---------
1) Failed to see the home page in HomeCept (tests/character/HomeCept.php)

Step  I don't see "Fatal error:"
 Fail  Failed asserting that   /
--> Fatal error: Broken! in ... on line ...
--> does not contain "fatal error:".

Scenario Steps:

2. $I->dontSee("Fatal error:")
 1. $I->amOnPage("/")

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
$
```

---

# Bug-Fixing Process

1. Fix the identified bug.
1. Run the characterization tests for all pages.
1. Some pages have *expected* failures as a result of fixing the bug.
1. Look for unexpected test failures, and fix the code as needed.
1. On the page(s) with an expected change, update the test(s) to match the new behavior.
1. Run the characterization tests again to make sure everything passes.
1. Commit, push, etc.

---

# You Need A Lot Of Tests

- Not enough to have one test for the one bug being fixed
- Spaghetti/global state means *any* page might get a new bug
- At least one for each browsable page (URL)
- How to build a suite for most or all of the application?

---

# Building A Baseline

- Use `wget` to spider the application

```bash
wget --recursive –-spider --domains example.dev http://example.dev
```

- Results in one file per URL
- Convert the files to characterization tests
- `see()` positive output, `dontSee()` negative output

---

# Tests Can Be Tricky To Write

- Activate the existing behaviors on each page
- "Success" can be tough to determine
- Minimum is straightforward
    - Fail on "error/warning/notice"
    - Pass on page-specific text
- Thoroughness just as hard as with unit tests

---

# Conclusion

- Characterization tests: "as it is" not "as it ought to be"
- Use Codeception "acceptance" tests with PhantomJS
- Need a baseline that covers most or all pages
- Process for how to use characterization when fixing bugs
- Tedious, time-consuming, detail-oriented
- Adds to certainty when making changes to legacy applications

---

# Thanks!

- @pmjones
- http://paul-m-jones.com
- https://leanpub.com/mlaphp
- https://joind.in/15189