Paul M. Jones

Don't listen to the crowd, they say "jump."

Hacking, Refactoring, Rewriting, and Technical Debt

(Not so much "lessons" here, as "observations and recollections.")

I.

From ESR, How To Learn Hacking:

  • Hacking is done on open source. Today, hacking skills are the individual micro-level of what is called “open source development” at the social macro-level. [2] A programmer working in the hacking style expects and readily uses peer review of source code by others to supplement and amplify his or her individual ability.
  • Hacking is lightweight and exploratory. Rigid procedures and elaborate a-priori specifications have no place in hacking; instead, the tendency is try-it-and-find-out with a rapid release tempo.
  • Hacking places a high value on modularity and reuse. In the hacking style, you try hard never to write a piece of code that can only be used once. You bias towards making general tools or libraries that can be specialized into what you want by freezing some arguments/variables or supplying a context.
  • Hacking favors scrap-and-rebuild over patch-and-extend. An essential part of hacking is ruthlessly throwing away code that has become overcomplicated or crufty, no matter how much time you have invested in it.

This strikes me as the opposite of, or at best orthogonal to, the practice of refactoring a large and long-existing codebase, especially that last. "Scrap and rebuild" is essentially "rewrite from scratch" and that is almost always a losing proposition in terms of time and budget.

But then, the description also says hacking is "lightweight, exploratory, modular" -- so perhaps it is possible to hack at components within a larger or critical system, and then refactor the system using that result. Cf. Hacking and Refactoring, also from ESR.

II.

I used to work for a consulting company with One Big Client (and a few smaller ones). This One Big Client generated a staggering amount of money; its business model was such that even a few minutes of downtime would result in large losses (technically "lost revenue" but the effect is the same).

The consultancy's director of development for the One Big Company didn't "believe" in technical debt. Or rather, he thought technical debt was a non-issue. He would write, or demand writing, some of the most horrific code, as long as it was written *quickly*, because the time constraints for the One Big Client were so overwhelming. There was no refactoring; he and his team would iterate fast, then wipe out the entire codebase and start all over again on a regular basis. (By "regular" I mean every 3-6 months). Any technical debt that was incurred was tiny in comparison to the monetary returns -- although it did take a toll on the programmers themselves.

Given ESR above, that's clearly a "hacking" approach to a critical codebase. Also, it's clear that *sometimes* a full-on rewrite might be the right thing to do, provided the right context. But I'm betting the vast majority of developers (I'm talking "five nines" here) are not operating in the context of that One Big Client and that particular consultancy. These were "skilled developers knowingly writing low-quality code" -- not to be confused with "*un*skilled developers *un*knowingly writing low-quality code." (Cf. Mehdi Khalili: "On Bad Code.")


How Terrible Code Gets Written By Perfectly Sane People

From Christian Maioli Mackeprang, the following:

Legacy code can be nasty, but I’ve been programming for 15 years and only a couple of times had I seen something like this. The authors had created their own framework, and it was a perfect storm of anti-patterns. …

And yet, it was not the code’s dismal quality that piqued my interest and led me to write this article. What I discovered after some months working there, was that the authors were actually an experienced group of senior developers with good technical skills. What could lead a team of competent developers to produce and actually deliver something like this?

I don’t necessarily buy all of these, but they’re worth thinking about:

  • Giving excessive importance to estimates: “Your developers might choose to over-promise instead, and then skip important work such as thinking about architectural problems, or how to automate tasks, in order to meet an unrealistic deadline.”

  • Giving no importance to project knowledge: “If you’re in a large project and there are modules for which you have no expert, no-one to ask about, that’s a big red flag.”

  • Focusing on poor metrics such as “issues closed” or “commits per day”: “When a measure becomes a target, it ceases to be a good measure. (Goodhart’s law)”

  • Assuming that good process fixes bad people: “[Fix your hiring.] Talent makes up for any other inefficiency in your team.”

  • Ignoring proven practices such as code reviews and unit testing: “Refraining from using tools such as modern IDEs, version control and code inspection will set you back tremendously.”

  • Hiring developers with no “people” skills: “It doesn’t matter that the other guy might be ten thousand miles away. One Skype call can turn a long coding marathon into a five-minute fix.”

Conclusion? “What you can’t assume is that just because you’ve signed up to apply Agile or some other tool, that nothing else matters and things will sort themselves out.”



Should the race go to the swift? Or should there be no race?

The race is not to the swift or the battle to the strong, nor does food come to the wise or wealth to the brilliant or favor to the learned; but time and chance happen to them all.

http://biblehub.com/ecclesiastes/9-11.htm

Some read this, and think:

  • The race should be to the swift.
  • The battle should be to the strong.
  • The wise should receive food.
  • The brilliant should be rich.
  • The learned should draw favor.

But others think:

  • There should be no swift and no sluggish; there should be no race.
  • There should be no strong and no weak; there should be no battle.
  • There should be no wise and no foolish; food should be given to all.
  • There should be no brilliant and no dull; all should be rich.
  • There should be no learned and no ignorant; all should draw favor.

The first is a cry against the vicissitudes of life. The second is a cry against reality itself.


Are people with very high IQs generally happy?

When someone is excluded from a group, ousted or ostracized, it makes him or her unhappy. Loneliness and exclusion is the most common reason for anomie - losing the reason for life and will to live and to commit suicide.

And overtly high IQ is the most certain way to get excluded and ousted. The reason is the communication range. It makes you different and not fitting in.

The concept of communication range was established by Leta Hollingworth. It is +/- 2 standard deviations (roughly 30 points) up or down on one’s own IQ. It denotes the range where meaningful interaction (communication, discussion, conversation and socializing) is possible. If the IQ difference between two persons is more than 30 points, the communication breaks up. The higher IQ person will look like an incomprehensible nerd and the lower IQ as a moronic dullard - and they will not find anything common.

+/- 30 points does not sound much, but once the IQ is past 135, the downsides are imminent. When someone has a perfectly mediocre IQ (100 for Caucasian average), his communication range is from IQ 70 to IQ 130, which covers some 98% of the whole population. But when it is 135, it is from 105 to 165, which is approximately 36% of population. And it gets worse: if it is 162, your whole meaningful set of human interactions is restricted to Mensa qualifying people only (2% of whole population). Good luck for finding friends, acquaintances, colleagues - or spouse.

And it gets worse.

When the average IQ of a group is lower than the lower end of your communication range, the group will see you as a hostile outsider. They will do anything to bully you out of their presence. They will ostracize, excommunicate and oust you amongst themselves.

Source: Susanna Viljanen's answer to Are people with very high IQs generally happy? - Quora


Original MVC Resources from Reenskaug

This is from Trygve Reenskaug: MVC: Xerox PARC 1978-79. Some quotes:

The essential purpose of MVC is to bridge the gap between the human user's mental model and the digital model that exists in the computer. The ideal MVC solution supports the user illusion of seeing and manipulating the domain information directly. The structure is useful if the user needs to see the same model element simultaneously in different contexts and/or from different viewpoints.

...

MVC was conceived as a general solution to the problem of users controlling a large and complex data set. The hardest part was to hit upon good names for the different architectural components. Model-View-Editor was the first set.

After long discussions, particularly with Adele Goldberg, we ended with the terms Model-View-Controller.

...

The MVC problem has more facets than I realized in 1979. I started working on a pattern language to disentangle the different aspects, that last draft was dated August 20, 2003. The plan was that it should be improved by a group of authors, not just the current single one. Unfortunately, the projct died at his point.


Reactionary Philosophy In An Enormous, Planet-Sized Nutshell

If you “criticize” society by telling it to keep doing exactly what it’s doing only much much more so, society recognizes you as an ally and rewards you for being a “bold iconoclast” or “having brave and revolutionary new ideas” or whatever. It’s only when you tell them something they actually don’t want to hear that you get in trouble.

Western society has been moving gradually further to the left for the past several hundred years at least. It went from divine right of kings to constutitional monarchy to libertarian democracy to federal democracy to New Deal democracy through the civil rights movement to social democracy to ???. If you catch up to society as it’s pushing leftward and say “Hey guys, I think we should go leftward even faster! Two times faster! No, fifty times faster!”, society will call you a bold revolutionary iconoclast and give you a professorship.

If you start suggesting maybe it should switch directions and move the direction opposite the one the engine is pointed, then you might have a bad time.

Source: Reactionary Philosophy In An Enormous, Planet-Sized Nutshell | Slate Star Codex


Murder Is An Urban, Not Nationwide, Problem

In 2014, the most recent year that a county level breakdown is available, 54 percent of counties (with 11 percent of the population) had no murders. 69 percent of counties had no more than one murder, and about 20 percent of the population and only 4 percent of all murders in the country.

The worst 1 percent of counties have 19 percent of the population and 37 percent of the murders in 2014. The worst 2 percent of counties contain 47 percent of the population and accounted for 51 percent of the murders. 68 percent of the murders occurred in only 5 percent of counties.

... To put it simply, murder isn’t a nationwide problem; it’s a problem in a very small set of urban areas.

Source: Murder Isn't a Nationwide Problem | RealClearPolicy


Slim and Action-Domain-Responder

I’ve had a warm place in my heart for Slim for a long time, and especially so since recognizing the Action-Domain-Responder pattern. In this post, I’ll show how to refactor the Slim tutorial application to ADR.

One nice thing about Slim (and most other HTTP user interface frameworks) is that they are already “action” oriented. That is, their routers do not presume a controller class with many action methods. Instead, they presume an action closure or a single-action invokable class. So the Action part of Action-Domain-Responder already exists for Slim. All that is needed is to pull extraneous bits out of the Actions, to more clearly separate their behaviors from Domain and the Responder behaviors.

I.

Let’s begin by extracting the Domain logic. In the original tutorial, the Actions use two data-source mappers directly, and embed some business logic as well. We can create a Service Layer class called TicketService and move those operations from the Actions into the Domain. Doing so gives us this class:

<?php
class TicketService
{
    protected $ticket_mapper;
    protected $component_mapper;

    public function __construct(
        TicketMapper $ticket_mapper,
        ComponentMapper $component_mapper
    ) {
        $this->ticket_mapper = $ticket_mapper;
        $this->component_mapper = $component_mapper;
    }

    public function getTickets()
    {
        return $this->ticket_mapper->getTickets();
    }

    public function getComponents()
    {
        return $this->component_mapper->getComponents();
    }

    public function getTicketById($ticket_id)
    {
        $ticket_id = (int) $ticket_id;
        return $this->ticket_mapper->getTicketById($ticket_id);
    }

    public function createTicket($data)
    {
        $component_id = (int) $data['component'];
        $component = $this->component_mapper->getComponentById($component_id);

        $ticket_data = [];
        $ticket_data['title'] = filter_var(
            $data['title'],
            FILTER_SANITIZE_STRING
        );
        $ticket_data['description'] = filter_var(
            $data['description'],
            FILTER_SANITIZE_STRING
        );
        $ticket_data['component'] = $component->getName();

        $ticket = new TicketEntity($ticket_data);
        $this->ticket_mapper->save($ticket);
        return $ticket;
    }
}
?>

We create a container object for it in index.php like so:

<?php
$container['ticket_service'] = function ($c) {
    return new TicketService(
        new TicketMapper($c['db']),
        new ComponentMapper($c['db'])
    );
};
?>

And now the Actions can use the TicketService instead of performing domain logic directly:

<?php
$app->get('/tickets', function (Request $request, Response $response) {
    $this->logger->addInfo("Ticket list");
    $tickets = $this->ticket_service->getTickets();
    $response = $this->view->render(
        $response,
        "tickets.phtml",
        ["tickets" => $tickets, "router" => $this->router]
    );
    return $response;
});

$app->get('/ticket/new', function (Request $request, Response $response) {
    $components = $this->ticket_service->getComponents();
    $response = $this->view->render(
        $response,
        "ticketadd.phtml",
        ["components" => $components]
    );
    return $response;
});

$app->post('/ticket/new', function (Request $request, Response $response) {
    $data = $request->getParsedBody();
    $this->ticket_service->createTicket($data);
    $response = $response->withRedirect("/tickets");
    return $response;
});

$app->get('/ticket/{id}', function (Request $request, Response $response, $args) {
    $ticket = $this->ticket_service->getTicketById($args['id']);
    $response = $this->view->render(
        $response,
        "ticketdetail.phtml",
        ["ticket" => $ticket]
    );
    return $response;
})->setName('ticket-detail');
?>

One benefit here is that we can now test the domain activities separately from the actions. We can begin to do something more like integration testing, even unit testing, instead of end-to-end system testing.

II.

In the case of the tutorial application, the presentation work is so straightforward as to not require a separate Responder for each action. A relaxed variation of a Responder layer is perfectly suitable in this simple case, one where each Action uses a different method on a common Responder.

Extracting the presentation work to a separate Responder, so that response-building is completely removed from the Action, looks like this:

<?php

use Psr\Http\Message\ResponseInterface as Response;
use Slim\Views\PhpRenderer;

class TicketResponder
{
    protected $view;

    public function __construct(PhpRenderer $view)
    {
        $this->view = $view;
    }

    public function index(Response $response, array $data)
    {
        return $this->view->render(
            $response,
            "tickets.phtml",
            $data
        );
    }

    public function detail(Response $response, array $data)
    {
        return $this->view->render(
            $response,
            "ticketdetail.phtml",
            $data
        );
    }

    public function add(Response $response, array $data)
    {
        return $this->view->render(
            $response,
            "ticketadd.phtml",
            $data
        );
    }

    public function create(Response $response)
    {
        return $response->withRedirect("/tickets");
    }
}
?>

We can then add the TicketResponder object to the container in index.php:

<?php
$container['ticket_responder'] = function ($c) {
    return new TicketResponder($c['view']);
};
?>

And finally we can refer to the Responder, instead of just the template system, in the Actions:

<?php
$app->get('/tickets', function (Request $request, Response $response) {
    $this->logger->addInfo("Ticket list");
    $tickets = $this->ticket_service->getTickets();
    return $this->ticket_responder->index(
        $response,
        ["tickets" => $tickets, "router" => $this->router]
    );
});

$app->get('/ticket/new', function (Request $request, Response $response) {
    $components = $this->ticket_service->getComponents();
    return $this->ticket_responder->add(
        $response,
        ["components" => $components]
    );
});

$app->post('/ticket/new', function (Request $request, Response $response) {
    $data = $request->getParsedBody();
    $this->ticket_service->createTicket($data);
    return $this->ticket_responder->create($response);
});

$app->get('/ticket/{id}', function (Request $request, Response $response, $args) {
    $ticket = $this->ticket_service->getTicketById($args['id']);
    return $this->ticket_responder->detail(
        $response,
        ["ticket" => $ticket]
    );
})->setName('ticket-detail');
?>

Now we can test the response-building work separately from the domain work.

Some notes:

Putting all the response-building in a single class with multiple methods, especially for simple cases like this tutorial, is fine to start with. For ADR, is not strictly necessary to have one Responder for each Action. What is necessary is to extract the response-building concerns out of the Action.

But as the presentation logic complexity increases (content-type negotiation? status headers? etc.), and as dependencies become different for each kind of response being built, you will want to have a Responder for each Action.

Alternatively, you might stick with a single Responder, but reduce its interface to a single method. In that case, you may find that using a Domain Payload (instead of “naked” domain results) has some significant benefits.

III.

At this point, the Slim tutorial application has been converted to ADR. We have separated the domain logic to a TicketService, and the presentation logic to a TicketResponder. And it’s easy to see how each Action does pretty much the same thing:

  • Marshals input and passes it into the Domain
  • Gets back a result from the Domain and passes it to the Responder
  • Invokes the Responder so it can build and return the Response

Now, for a simple case like this, using ADR (or even webbishy MVC) might seem like overkill. But simple cases become complex quickly, and this simple case shows how the ADR separation-of-concerns can be applied as a Slim-based application increases in complexity.


Why MVC doesn't fit the web

[MVC is] a particular way to break up the responsibilities of parts of a graphical user interface application. One of the prototypical examples is a CAD application: models are the objects being drawn, in the abstract: models of mechanical parts, architectural elevations, whatever the subject of the particular application and use is. The “Views” are windows, rendering a particular view of that object. There might be several views of a three-dimensional part from different angles while the user is working. What’s left is the controller, which is a central place to collect actions the user is performing: key input, the mouse clicks, commands entered.

The responsibility goes something like “controller updates model, model signals that it’s been updated, view re-renders”.

This leaves the model relatively unencumbered by the design of whatever system it’s being displayed on, and lets the part of the software revolving around the concepts the model involves stay relatively pure in that domain. Measurements of parts in millimeters, not pixels; cylinders and cogs, rather than lines and z-buffers for display.

The View stays unidirectional: it gets the signal to update, it reads the state from the model and displays the updated view.

The controller even is pretty disciplined and takes input and makes it into definite commands and updates to the models.

Now if you’re wondering how this fits into a web server, you’re probably wondering the same thing I wondered for a long time. The pattern doesn’t fit.

Read the rest at Why MVC doesn't fit the web.