The words any software developer dreads hearing are “Can you just change this one thing for this one user/client/company”. The requests usually have perfectly valid reasons, but it’s sometimes hard to explain to the person asking for the feature why it’s more work than just the code. Most of the time it’s not the coding that’s terribly hard, it’s making it just so for that particular user, testing it, maintaining it and then you have the nightmare two years down the line when you have to change code that touches it.
Much has been written about system complexity but it tends to read like this:
Great for your PhD but terrible as a demonstration/case study.
Hopefully, the following little story can serve as a way of visualising the pitfalls when a system ends up in an ever increasing cycle of complexity.
In The Beginning: A Village Fete Tea Stand
Agnes, Beatrice, Clive and Doris decide to run a small tea stand at the village fete in their home village of Systemton in the Yorkshire Dales. They do a quick run to Costco for a giant box of Yorkshire Tea and some huge cartons of milk, Agnes borrows a tea urn from the village hall, Clive pops into IKEA for 50 cheap tea cups and a handful of teapots and they’re all set. They decide on a nice round 20p per cup to make charging customers very simple. Doris will brew the tea, Beatrice will take the cash, Clive will collect the used cups and Agnes will wash them up. They knock up a quick sign that says “Tea, 20p” and hang it outside.
Very soon, they’re running flat out and things aren’t running smoothly. To cut down on the wait they’re pre making the tea, but it’s hard to keep track of how long each pot has been brewing. Some customers are complaining it’s stewed and some are having lightly coloured water. Also, some customers are complaining it’s too milky. It seems that milk is being added to make the colour the same, but that means lightly brewed tea is only getting a splash, whereas over brewed tea is getting far too much milk.
Being an ex RAF engineer, Clive decides it’s time for a system. He labels the teapots A, B, C, D and E and knocks up a chart. Using his watch he carefully notes the time each pot is filled and removes the teabags after exactly 5 minutes. They can adapt to demand by having between 1 and 5 pots on the go at once meaning the tea stays hot. He also finds a measuring cup that looks about right and milk is poured into that first before going into the cup, so each cup has exactly the same amount.
After 30 minutes or so Agnes overhears some friends say it was a nice refreshing cuppa, but perhaps a bit on the strong side. She feeds this back to Doris who adjusts the brewing time down to 4 minutes. Feedback seems to be much better from everyone and they settle on 4 minutes as the ideal brew time. They now have a system for the perfect cup of Yorkshire, word gets around about how good the tea is, and they collect 20p per cup from happy customers all day long. In fact most customers not only have 20p ready, most of them pop it in the box themselves making life very easy indeed for Beatrice so she can help Doris with the tea making.
Year 2: More Options
It’s fete time again, and this year Beatrice accompanies Clive on the Costco run. As well as the many jugs of semi-skimmed and the box of Yorkshire Tea bags, Beatrice puts a box of her preferred PG Tips in the trolley. “Good idea” says Clive.
Fete day comes, the stall is running like clockwork with Doris using Clive’s system from last year. She also had a brainwave and brought along 5 kitchen timers and she sets them off when each pot is filled. When the timer beeps, she takes the teabag out of the relevant pot and bingo, the perfect cup of tea.
Except there’s a snag. They have both boxes of tea on display behind the stall, and both are being offered to customers. Doris tries to adapt to the new option by having pots of Yorkshire and a pot of PG Tips on the go, she just makes pots A, B, C and D Yorkshire and E is brewed with the less popular PG Tips. Everything goes fine until Beatrice has a cup of tea herself and chooses her favourite PG. Yuck.. over brewed. She feeds back to Doris that PG Tips only takes 3 minutes to brew. Doris adjusts her countdown timer for pot E to 3 minutes and they now have perfect cups of tea in two flavours. 20p’s are collected all afternoon and word gets out about the perfect cuppas.
Late in the afternoon after most locals had left, a coach stops off at the fete. As they’re paying their entrance fee, Agnes has the idea to get all the teapots brewing in preparation. Four pots of Yorkshire and a pot of PG on the go. The first customers appear and order PG. As do the other 50 from the coach. Oh dear. This coach was from Lancashire and they don’t want the Yorkshire. Doris has to suspend brewing of the 4 pots of Yorkshire throw them away and restart with PG. Halfway through she remembers that the 4 timers were set for 4 mins not 3. Hastily she changes the process so they don’t stew.
Still, crisis largely averted, the coach load pay their 20p’s, our four friends collect the money and sit down for a well earned rest!
Year 3: Demand Increases and Scaling is Required.
The four friends dust off their IKEA mugs and teapots, do a Costco run again, pick up the milk, the Yorkshire and the PG
Fete day arrives and they’re blessed with glorious weather, so much so that the village is overrun with visitors from the surrounding area eager to see the ever expanding Systemton Fete. Agnes, Beatrice and Clive are rushed off their feet collecting/washing cups to re-use whilst Doris makes the tea. The system of multiple teapots works well in general and they sort of adapt by having a split of 4-1 or 3-2 depending on demand, not forgetting to change the timers each time of course!
Lunchtime comes and the queue is out the door, there’s simply not enough cups or teapots to scale to the required throughput. Agnes asks her friend Edith to rustle up a load more cups from peoples houses as well as two more teapots. Brewing speeds up again and the queue decreases. However, not all is rosy. Customers from years before have come to expect the perfect cup of tea and it seems some are getting overly milky or stewed tea again. What on earth is going on? They have a quick internal management meeting and the problem is obvious. In their attempt to scale to demand they introduced different sized teapots and cups. The 4min/3min times and the measure of milk no longer work. The system is broken. Their only options now are:
- Scale back down, make customers wait for the perfect cup.
- Supply poor quality tea to a percentage of customers, but quickly.
- Try to modify the process with some different variables for the different sized pots and cups which will require extra man power.
Clive stops collecting cups takes one of the IKEA teapots and the two ‘new’ ones (therefore massively reducing productions whilst they’re offline). He measures the volume of each and he decides that because they’re 50% and 150% the size of the IKEA ones, brew times should be:
Yorkshire PG Tips Ikea Pot 4 mins 3 mins Large Pot 6 mins 4.5 mins Small Pot 2 mins 1.5 mins
He does a quick test with each pot using PG (as it takes less time than to test Yorkshire) and it all seems ok.
He jots the times down and hands it to Doris who rolls her eyes at him. She asks about the milk measure and the different sized cups. They look around at their customers… there’s the 50 identical cups and 50 more borrowed cups each of which a completely different size to one another. Whilst possible, the work involved to measure and test each cup would be horrendous, not to mention from an operational perspective doing different measures. Doris has more than enough to do without different milk measures for 51 different types of cup. If they choose to go down this path they’ll need a dedicated person just for milk measures (who, incidentally, won’t be needed when it gets quieter and they can drop back to the 50 original cups).
Milk measures aside, they’re now running at a huge scale, supplying at least twice as much tea as before. They’ve had to rope in Edith to help Doris of course as managing the various combinations of tea type/pot size is unmanageable for one person. In fact, it’s getting too much for the pair of them when demand changes and they need to swap to say 3 PG Tips pots, but Beatrice can help out as she doesn’t need to be looking after the payment full time as it’s simply 20p per customer and 9 out of 10 customers put a coin in the box themselves anyway.
Year 4: Requirements By Committee and Homemade Cake
This year, Mavis, the head of the village committee decides she like the tea stand and it should have her input. She elects herself into the team as Tea Stand CEO, and decides that they need to be offering a few more exotic teas such as Darjeeling, Assam and Jasmine. And biscuits.
The news is broken to the rest of the group on the Friday before the fete. “Oh dear”, thinks Clive as off he goes in search of tea. He can get PG and Yorkshire (and milk and biscuits) from Costco, but the others require a search. Eventually he finds the others but at a much much higher cost. “Hmmm, not sure Mavis thought about the cost of these fancy teabags, but we’ll have to work something out”
He gets home and calls the others. In preparation for tomorrow they decide not to get caught out this time and they test the new teas beforehand and they work out what tastes right. All testing is carried out with the standard IKEA pot.
Tea Brew Time Yorkshire 4 mins PG 3 mins Darjeeling 2 mins Assam 3.5 mins Jasmine 5 mins
Great. Except they’ve not had any budget to increase their capacity of IKEA teapots (though fortunately, they did manage to buy 100 more cups so at least those are now standard) so there’s still this issue of the two other pots. Using the same method as last time they do some guestimates and hope the values are right.
Yorkshire PG Tips Darjeeling Assam Jasmine Ikea Pot 4 mins 3 mins 2 mins 3.5 mins 5 mins Large Pot 6 mins 4.5 mins 3 mins 5.25 mins 7.5 mins Small Pot 2 mins 1.5 mins 1 min 1.75 mins 2.5 mins
This chart is written out, and they then turn their discussion to pricing. Working on the cost per teabag it’s decided that Yorkshire and PG are 20p/cup and the other 3 should be 30p/cup as 20p is just too low to make any money on the more expensive exotics. Of course, this is assuming the customer wants milk. 99% of PG/Yorkshire customers have milk, it’s pretty optional with Assam, but a no-no with Jasmine and Darjeeling. It’s decided that it’s unfair to charge all customers for milk, so actually the milk should be charged at 2p/measure, so the prices are adjusted to 18p and 28p with a 2p extra if you have milk.
Of course there’s also the biscuits which are charged at an easy 10p each regardless of biscuit (some will end up paying over the odds for very small biscuits from the selection box, but that’s life)
They throw away their sign from year 1 and make a new one.
Fete day arrives again, and it’s quickly apparent there’s a problem. Customers are asking lots of questions before placing the order. Some are unsure about the different tastes of the fancy teas. With Doris and Edith brewing, Beatrice on the till, Clive collecting, Agnes washing and Mavis doing…. CEO things, there’s nobody to answer customer queries, so they enlist the help of Florence who is given some training in Tea and put at the front of the stall to field questions.
Within half an hour, brewing operations has fallen apart as well. Beatrice is too busy now with the more complicated money side of things and the serving of biscuits to help out like last year, leaving Doris and Edith not only having to deal with the Yorkshire/PG in different sized pots problem, but they have the other teas. Problem is, those teas aren’t being ordered in sufficient quantity to justify a pot on the go so they’re being made in the cups. Reacting to the change and altering the process by brewing in cups means of course the well previously researched pot times are all wrong for the special teas, so they have to rapidly re-test them in cups while in full production. It also means someone has to monitor the times of each cup. Doris and Edith decide they’ll look after the pots and they draft in Edith’s husband Graham to do the individual teas using the old style chart and watch method of timing the brew according to the pre-set brew times (there’s nowhere near enough kitchen timers). The gang muddle through, each working flat out to keep the production line running.
At lunchtime, Mavis starts chatting to her friend Olive who is running a home made cake stall. Olive lets on that she’s having a slow day. Ever one to help out a friend and maybe do some business along the way, Mavis has an idea. Why not re-sell Olive’s cakes on the tea stand. They’ll take the money and give Olive a 30% cut. Everyone’s a winner baby.
Mavis brings over a tray of each of Olive’s 20 different cakes complete with the price label on each cake, along with some plates and cutlery and places them on the stand. Unfortunately, this just serves to confuse customers more with all this choice and they soon turn to Florence to ask about the cakes as well as the tea. Florence though, being diabetic knows nothing about the cakes as it’s just not her area of expertise, she’s now a tea maestro though. She answers as best she can until Mavis grabs Olive’s daughter Heather to answer cake queries.
The end of the day arrives and our exhausted team take the proceeds, calculate Olives cut by using the sales ledger, split the remaining proceeds 9 ways between them and go home exhausted.
Year 5: The Final Chapter, The Team Go To India!
Word has finally got out about the original perfect cup of Yorkshire from year one, the product that got them their reputation. The team is invited to the World Tea Championships high up in the Indian Himalaya. The team do their shopping, buy some more standard teapots (no messing about this time) and pack their gear. The week before, they get together all 9 of them and revisit all the brewing times of the various teas, all in pots this time. After a lot of focus grouping amongst external friends and past customers they arrive at some very specific, perfect times, slightly improving on the original Yorkshire and the latter PG tips times as well as the fancy teas.
|Ikea Pot||4.1 mins||3.2 mins||2.4 mins||3.1 mins||4.3 mins|
Arriving in the Indian Himalaya the intrepid tea-makers trek high up into the hills to the venue for the competition. They now have a finely honed system, perfect timings, standardised cups/teapots, a dedicated workforce, many options of tea, Fox’s biscuits and Olive’s home made cakes. There’s no way they can lose this.
The big day arrives and the scene is hectic. There’s thousands of tea lovers from all over the world and our team get brewing. The first few customers don’t seem massively impressed. Still, undeterred they continue serving from the various pots, keen to bring Yorkshire and PG Tips to the world. Except all the customers are hating it. Could it be they’re not used to it? Our team try a cup of their finest Yorkshire….. and it’s terrible. Something has broken and the brew is off. Way off. The altitude of the venue is lowering the boiling point of water to 85degC, not even close to what they’re used to.
Our crew can either try to re-engineer their processes by trialling new brew times for each of the 5 teas, buying more pots (since brewing is longer, they’ll need more pots on the go, and the extended brew time makes it even harder to react to demand as the lead time is longer) and maybe train yet another staff member in brewing to keep up with capacity now that brewing is the bottleneck.
They ditch everything apart from their beloved Yorkshire, they trial a new brew time at the lower boiling point. They use ALL their teapots for Yorkshire, they ditch the separate charge for milk and put the cup price back to 20p. They average out the price on the cakes to another 20p to simplify billing. They give away biscuits to customers who bring friends along rather than trying to charge and complicate the pricing structure. All day long, they serve the perfect of tea to hundreds of happy customers, not one of which asks them if they can have Jasmine.
Adding increasing numbers of options, features, widgets and conditions leads to an ever increasing burden of complexity not just in building a system, but in designing, testing, supporting and don’t forget operationally. It’s not always the best experience for the customer either.
Our team didn’t quite end up where they started though. They added biscuits and delicious homemade cakes. They also productionized the tea making and refined the brew time. They also learnt a lot along the way.
Now, time to put the kettle on…