Pieter Nagel: Swapping Thoughts

[This is Part 2 of my Quality and Productivity series].

In the first instalment of my Quality and Productivity series, I introduced a particular way of looking at something like a software development project:

Instead of thinking of the project as a sequence of tasks to be completed, each of which costs a certain amount of time to complete, think of it this way: a project is a series of actions, each of which has short-term and long-term consequences. You can visualise these consequences as a green (positive) or red (negative) graphs with a certain shape (brief spikes, flat lines, or exponential curves) extending far off into the future.

This brings up an important corner case: none of what I’m trying to explore in this series has any relevance if the project is not intended to have any future at all. If the explicit plan is that the project should fail, or terminate in the near future, then by all means feel free to excuse yourself from this discussion.

Today, I want to play a bit with the conceptual vocabulary I’ve established. So, let’s combine some of our concepts:

In the above hypothetical scenario, a project took an action and derived some immediate short-term but short-lived benefit from it – the green spike. But the cost, or consequence of the benefit was an ongoing an increasing recurring detriment. Continuing in the same vein as in my last installment, one could call this “selling your soul to get the dishes done”. Don’t try this at home.

However, since no-one on earth would be so crazy as to do this (I hope), let’s move on to a potential permutation that’s a tad more realistic:

In this case, the same short-term benefit was gained, but this time at a much lower cost: specifically, a small but constant ongoing cost.

The red part of the curve is more realistic, since it’s not unheard of for things to have ongoing constant costs – if you employ someone, you have to repeatedly pay them a salary for the foreseeable future; if you add a feature to your codebase, you will incur a small but repeated cost to maintain that feature in the face of framework upgrades.

However, the green part is still a bit atypical: usually software is developed in order to gain an ongoing benefit, such as clients paying fees for the services the software enables.

But I first want to pause here a bit.

Readers may suspect by now that I’m getting at the topic of trade-offs of long-term vs. short-term costs and benefits. So, they may guess that I’m heading towards saying that ongoing long-term cost will eventually negate short-term immediate benefits. Well, I thought that I was driving towards that point.

So, let’s introduce another concept: You can think of the state of your project today by stacking up all the graphs of all consequences of all past actions. Projects don’t consist of single actions, they consist of a multitude of actions, week-by-week and day-by-day. Each action has its own consequences; each of those consequences could potentially be good or bad, short-lived or long-term.

Now let’s explore what would happen if one made a habit of this hypothetical business model of getting short-lived benefit at constant long-term cost:

Well, ok. So the ongoing costs keep getting bigger and bigger. But hey – they do manage to get a nice ka-ching! in the cash register at regular intervals.

It is at this point, I guess, that some astute readers are really getting irritated with me. What’s with the silly green and red distinction between costs and benefits? Why don’t I just simplify my model by adding them up and talking about nett costs and benefits?

I must admit that my inner mathematician made the exact same objection, and it took me a while to figure out why I instinctively wanted to keep the red and green separate. But I kept them, and I have my reasons.

But for now, let’s acquiesce, and replace the above with a nett graph. Better yet, let’s graph the cumulative costs and benefits accrued over the entire lifetime of the project:

Hey, that doesn’t look to bad! We’ve got a realistic cost model and a hypothetical pessimistic benefit estimate (only once-off short-term benefits), and even then the nett benefit seems to go through the ceiling. Maybe this model is sustainable after all!

Even I was stumped for a while, since my I set out to show that the hypothetical model is not sustainable.

And that’s where will pick up from in the next installment.

Agile Development

A friend of mine is saddled with occasionally maintaining some TCL code, because a few years ago a programmer who was a “TCL fanatic” worked at her company and then moved on. Neither she nor anyone else in the company really loves TCL, and so she understands it only enough to maintain the damn program as required. Sometimes she goes all wistful at the idea of rewriting it all in C, in that “wouldn’t it be great if – but it ain’t never going to happen – but one can dream…” tone of voice.

Today, I am not going to comment on how suitable either TCL or C is for the job. I will take it as a given that, TCL is the wrong tool for the job and that C is the right tool for her to use – in her specific context. Because, of course, she and whichever reasons make her prefer C is an important part of her specific context.

However, I do want to comment on the “rewrite” bit. Rewrites are fraught with risk and are almost invariably more complicated that anticipated. See, for example, Uncle Bob on The Big Redesign in the Sky.

I’ve lost count of the times when I’ve had to make a “small” change to our system which we thought would “obviously maintain existing semantics”, only to find that the big bold leap to the new algorithm doesn’t work. In those cases, it has been of great value to retain both the old and the desired new implementation simultaneously, add asserts to our code that verify that both algorithms yield the same result, run the whole test suite, and let the computer pinpoint the fallout. Fix test failures, lather, rinse repeat.

And all that in the context of a small modification within a single existing system! I shudder when I contemplate the notion of trying to create a whole second system from scratch.

Luckily, there are better ways than a blank-slate rewrite. One option is the “Strangler Pattern” Martin Fowler described (also know as the “Strangler Vine Pattern”). The basic idea is to wrap the new system around the old system, siphon off more and more of the external-world inputs to the new application, until the new application replaces the old. Somewhat like a strangler vine replaces the tree it parasitises for scaffolding.

But there is another alternative, which I would call the “Grandfather’s Axe” pattern:

As the proverb says: “This is my grandfather’s axe: my father fitted it with a new stock, and I have fitted it with a new head.”
—Robert Graves, The Golden Fleece, p. 445

If you replace all the parts of something one by one, is it still the same thing at the end? Or think of Theseus’ Ship of Greek legend, which slowly, over time, had all its’ planks replaced. Is it still the same ship?

Lost argonauts, recently spotted in the present-day incarnation of Theseus' ship.

Instead of building a new system around the old one, replace parts of the old system one by one until only the new system remains. In the Strangler Pattern, the new system gets control before the old system does, and in that way intercepts more and more functionality. In this pattern, each part of the new system gets control at exactly the same point as the old part did, and functions within the context of the old system.

So, I would advise my friend:

1. Install something like SWIG.
2. Choose a simple, self-contained TCL function without dependencies to the rest of the TCL system. Reimplement it in C.
3. Figure out how to get the old TCL code to call the new C function.
4. With the mechanics of SWIG and TCL to C integration under your belt, next time you need to maintain a TCL function, try to move it to C first before extending its’ behaviour.
5. Repeat until no TCL is left. This is the crux: if there’s no will to see if through, one is left with a more complicated system than one started off with.
6. Discard the SWIG scaffolding.

One need not end up with a straightforward C port of the original system this way. As more and more of the system lives in the context you prefer, it should be easier to mould that part of the emerging new system into a new shape. That’s the premise: that the code is so much more expensive to maintain and modify in TCL that it would be cheaper and easier to modify it in C. If this premise isn’t true, there’s no point to this exercise.

Since the essence is to gain maintainability, I would also strongly advise taking the opportunity to explore Test Driven Development to bring the nascent new system under automated testing and gain a better handle on maintaining it.

However, I do feel compelled to point out that personally, I would never choose to move towards C in this way. I would use this technique to move away from C toward something like Python, Ruby, Lua, Scala or the like. Or, if appropriate, I would use this technique to isolate the speed-critical code into C subcomponents and the rest into a more malleable higher-level language – also known as the Alternate Hard and Soft Layers pattern.

Agile Development

[This is Part 1 of my Quality and Productivity series].

For the most part, we tend to think about software projects as a sequence of tasks: we need to build the Whizz-Bang Widget, then the Decimal Doodad, then the Super-Service Subsystem.

Tasks have costs associated with them, for example monetary costs (often mostly from salaries), and time costs (from the passage of time as the programmers sit around earning those salaries while doing whatever on earth is is they do when they create those doodads). Both of these costs are linked to time.

If we want to become a bit more sophisticated, we can say that tasks have inter-dependencies: we need to do the Widget before we can start tackling the Super-Service. But mostly, the concern here is once again time: task B will have to wait for task A, which means time has to pass.

And for the most part, it seems to me like our industry acts as that what it’s all about: things need to be done, and preferably quickly. Most of of our project management meetings, and artifacts, and Gantt charts, and status updates, and Burn-down charts, and task estimation, most of our de-facto behaviour that an outside anthropologist would notice – it all seems to me to be focused on time: when can I have this done?

I’ll call that task-based thinking, and my aim here is to break that mold a bit.

For a start, not all “tasks” are the same. Over the past few years I started sorting tasks, in my mind, into categories which I’ll call Dishwashing, Coin-dropping, Investing, and Hemorrhaging.

The essence of the distinction I want to make is this: tasks have certain consequences – either good, or bad (or both), and these consequences play out over time, long after the task is done. I imagine those consequences as a graph with green or red lines, going off into the future.

Dishwashing

You could do the dishes sloppily, with a half-hearted rinse and wipe, or you could scrub them up like a surgeon washes his hands. But for the most part, once you’ve used the dishes again and washed them again, no trace of how well or how badly you washed them the previous time remains.

The shape, over time, of the consequences of Dishwashing type tasks are pretty boring. There’s a brief blip of goodness as the plates gleam and the kitchen is tidy, but by tomorrow the value of that single act of dishwashing is gone, for ever:

Dishwashing: value over time

An example of this in the programming world would be: you are developing on your system, and the app suddenly crashes. You guess it’s “that bug” again, the one that (you hope!) only surfaces in the development environment. So, instead of permitting yourself to get “sidetracked” by figuring out the source of the bug, you just do the proverbial reboot, or “make clean; make" so that you can focus on the task at hand again because you know by experience that will make the bug go away – for now.

I would call that type of thing dishwashing. Rebooting helped you now, but it did absolutely nothing to prevent “that bug” from biting you in the backside again tomorrow, and the day after that, and next week, and the week thereafter…

I saw another good example of dishwashing this week: I had a mole cut out, and afterwards the doctor sat in front of her laptop for 15 minutes to make up the account. First she had to add a line item for “consultation”, then for “anaesthetic”, then for “stitches”, then for “wound dressing”, then for “sterilising the pincettes”, then for “disposing of the scalpel blade”, and so forth. As she did this she apologized profusely: their accounting system can actually cope with adding categories like “mole removal” that break down into typical line items, so that they need only add the unexpected items to the account. But, she said, they haven’t had the time to get round to setting it up yet. Well, what she did was dishwasing: it helped to get my account done, but did nothing to help get the next patient’s account done.

Investing

Years ago you could have taken take a sum of money and invested it in Google; or you could have put the same money in Enron. A week later, you wouldn’t have noticed much of a difference, but by now you’d have noticed that the act of investing in Google just keeps on giving, whilst the act of investing in Enron gave you nothing.

The time-shape of Investing type taks is more interesting. The value may take a while to kick in, but thereafter it remains a constant (in this case, literally) presence in your life:

Investing: value over time (constant)

In the financial world, that’s a pretty atypical investment, one with constant returns. But examples of this in the programming world are not so rare: Imagine you make an improvement that shaves a minute off your total compile time, and imagine you do a full compile five times each day, on average. Then your act yields a constant daily 5 minute return on investment for each and every single day thereafter.

But, of course, non-linear return curves are also possible, and not just in the financial world:

Investing: value over time (exponential)

What would an example of this be in our industry? Imagine you run the some sort of online service, and you make your money off subscriptions. You spend a certain fixed amount of time to release an API which allows other people to write plug-in apps for your service. Imagine that causes a community form around your service, and every day the word of mouth spreads to more and more people, who subscribe to your service to make use of the cool plugins their friends are nattering on about. And then, the next day, they natter on about it to two other friends, and so forth,

Oh, and by the way: when you look at these graphs you may wonder what measure of time is represented by the X axis. Well, for now, my answer is: more than you think. If you thought I drew a graph of how the consequences of the action panned out over the next week, no. I drew the next month. If you thought I drew a month, no. It was a year. If you thought I drew a year, well, actually, you’re looking at a decade.

Coin-dropping

Thus far, I’ve only drawn green graphs. Now it’s time for the red.

If, for example, you accidentally drop a coin on the ground when you peek inside your wallet, that would be a loss, but at least when it’s over, it’s over. So coin-dropping is the mirror image of dishwashing:

Coin-dropping: cost over time

IT example: you lean back in your chair and accidentally kick the power switch. Computer off. Dang.

Or, you hit ALT-TAB once to many and get distracted a single session of reading Slashdot.

Hemorrhaging

Hemorhaging is investing’s evil twin. Instead of keeping on giving, it just keeps on taking:

Hemorrhaging: cost over time (constant)

An example of that in the computer world would be if you make a change that causes your system’s test suite to run 1 minute longer than before. For each and every day after that, your project loses (number of developers) × (number of test runs) minutes.

But here comes the stuff of nightmares: You get your running account and your credit card account numbers confused, and thereafter, you buy the office stationery, pay your electricity bill, run your company’s payroll, everything off the credit card. Not because you desperately need to stay afloat for just one more month until that huge contract you did pays out, no: just by accident. And then you only notice the mistake once the interest payments exceed your income:

Hemorrhaging: cost over time (exponential)

What would be an example of that in the development world? Well, for example, you choose to add something to your toolchain whose running time is O(n2) proportional to number of lines of code in your system. Today things are okay, but next quarter development is a bit slower, and next year ever worse, and five years later you institute a system where each developer has 30 PCs and rotates amongst them each day, so that she can continue coding on the changes she started compiling last month.

Agile Development

This blog post is a placeholder index for a whole series of posts which I wish to do in future on quality and productivity, especially in the context of software development. I will update it to link to each future installment as I make it.

Over the past ten years I have been working as part of a team of developers who work on a single software system.Ten years is not exceptionally long, but I do suspect that it is much longer than the average time most programmers remain involved on a project.

In that time, I’ve gained some experience that I feel has given me a different perspective about many of the decisions and trade-offs we make day to day- because I’ve seen the long-term effect of many such decisions.

In short: I vociferously agree with the Agile and Lean dictum that to increase productivity, one needs to increase quality. Or, the converse: that cutting corners and lowering standards in order to get work done faster is a surefire recipe for decreasing productivity – bogging it down..

Nothing new there; nothing that W. Edwards Deming, the Lean movement, Kent Beck, the signatories of the Agile Manifesto, and may practitioners in my own and other industries haven’t been saying in their own way for the past few decades.

But I find that I’ve grown to think about these things in ways that make it difficult to communicate with people who don’t share the same mental framework. More importantly: I find myself curious about what exactly my own mental framework is: I write it down for my own discovery more than anything else. In my mind, it feels like what I will write in future will be something akin to deriving a lot of the values and rules of thumb of the Agile world “from first principles”, as’t were, because it all “clicks together” in my head in a way I wish to share.

The outline as I anticipate it so far:

• Part 1. Dishwashing, investing, and hemorrhaging.
• Part 2. Selling your soul to get the dishes done.
• The fallacy of “task-centric” thinking.
• Ticking off a to do list, or making repeating appointments?
• Cognitive bias: project mistakes we’re hard-wired to make
• Over time, most systems tend to become absolutely dominated by long-term negative effects of short-term decisions.

Agile Development