Mallorca 70.3 Race Report

This is my first race report. It’s huge (like medium sized novel huge). I’ve gone into far too much detail, but that’s entirely for my own benefit so I can remember it all in years to come,  so apologies in advance.

2014 Training and preamble

I’d booked Mallorca almost a year ago after having just completed a ‘novice’ super sprint triathlon. I wanted something to work towards, something hard but achievable with the right work. Make no mistake, for me, a 70.3 distance (1.9km swim, 90km bike, 21km run if you happen to be one of the 3 non tri people reading) is no joke for me. Some will knock that out as a tough training day, but for myself, with no real athletic background, it had sat on my wall planner like an ever nearing panic attack. I’d done a sort of Olympic distance race before (swim was cancelled and replaced with a 5km run) and that wasn’t too bad, but this was going to be an entirely different ball game.

Training wise, 2014 was mixed. A lovely mild January meant I could actually go outside and learn how to run. I ran 3 times a week for about 5-7km at a time which for me was quite big volume. Between the start of Jan and the end of Feb my 5km tumbled from 27.50 to 23.55, which was incredibly encouraging. Running felt easy, everything was good. Bike time was still limited, but I quite enjoy the turbo trainer anyway and was making good progress on TrainerRoad working through the sessions.

Towards the end of February though, I think my body had had enough and decided to throw various leg/hip injuries at me. I had a complete week off and shelved the running entirely, concentrating on turbo and a lot of strength training. Late March I did the MK half marathon which was double the distance I’d ever run. Encouragingly it felt great and was on track for a respectable 2 hours until the last few miles when I started to get knee pain. Hobbled to a 2:25 finish, but job done. Every run for the next month though was plagued with knee and hip pain. I could manage about 5km but beyond that, no chance.

Swimming has come on leaps and bounds too thanks to Keith Lewis at the Swim Shed, transforming my stroke from salmon in fishing net into something resembling an efficient swimmer. With a somewhat varied block of training I boarded the plane to Mallorca with the sole aims of having a nice time and walking away with a medal.

Race Lead Up

Sunday – nice recce of the course from our rental van (don’t ask). The climb on the course seemed absurd but the rest of it seemed fine. Quite a long way, 50 odd miles on winding roads. Built my bike up in the evening and then went for a nice meal out

Cheers!

Cheers!

 

Monday – good play on the beach with Ben. We made some sandcastles and I walked out into the sea carrying him. Couldn’t get him to splash around though as he said the sea was scary. Later in the day, friend and amazing athlete Amy Kilpin asked if I wanted a bit of hand holding with my first sea swim. Yes please! The sea doesn’t “freak me out” as such but it did seem a bit vast to just go in on my own. We both also needed to test our bikes out so we did an easy (for her, not for me) ride up to Port de Pollenca and back. Riding on Mallorcan roads just felt faster for some reason. It was also the first test of the deep aero front wheel and rear disk cover with some serious side winds. Other than a tiny wobble now and again it felt fine.

Back home, to the beach and wetsuits on it was time to man up and get in the water. The calm millpond water from the morning was replaced by some relative chop and the odd occasional proper wave. Swimming out left me feeling totally breathless, I couldn’t get the timing right and I was really struggling. All the pool swimming I’d done had gone out of the window. Half an hour more swimming and some pointers from Amy, I was feeling a bit better. 1900m though was going to be tough. Best moment of the day though was walking out of the sea to see Ben up to his elbows in the water having a great time splashing around.

Tuesday – had an hours swimming on my own while Ben and Nat built sandcastles and played in the sea. Much much better than Monday. The water was still and I found the rhythm I’d been missing.

Wednesday – another morning sea swim, this one felt great. Managed to keep pace with a chap who turned out to be an OW coach from the UK. He gave me a couple more tips, watched me swim and said it looked good. Confidence restored! In the afternoon we picked Amy up and put the bikes in the van to do the descent. It was the one part of the ride I was worried about. It’s mega fast in places and incredibly technical along switchbacks in others. Turns out it was hilarious fun. Years of driving quick cars on track meant it came quite naturally. Brake late and if needed all the way in to the apex to keep the weight on the nose, use the full width of the road, let the brake off as you come out the corner and I made pretty good pace. Natalie followed us down and picked us up at the bottom. The drive home was a 2 hour adventure of diversions that involved driving through an olive grove and a car blocking the road with a sign that said “I’m here with the asphalt machine” as the locals were frantically resurfacing before Sunday.

Thursday – quick splash in the sea followed by event registration and collecting my transition bags. Holding the race number in my hand it all started to feel a bit real. Nerves starting to kick in now so spent the day playing with Ben to try to take my mind off it. Once he’d gone to bed I packed, repacked and triple packed my transition bags before an early night. Couldn’t sleep at all.

Friday – easy morning, another play on the beach with Ben, followed by bike racking. Had met up with Amy before bag drop to have someone sanity check my bags as I was convinced I’d forgotten something. Bike racked, bags dropped and a few technical issues sorted and was good to go to the briefing. Seeing hundreds of amazing looking athletes in one place did nothing to calm my nerves and feelings of being utterly out of my depth but the briefing itself was good. After a few hours of slow boiling in the sun though we were both starting to feel the effects of the heat. I always notice my speech starts slurring a touch if I’m overheating and it was just creeping in, so we went on a hunt for some ice cream (nutritional and medicinal, honest) and some shade. Nerves still at full tilt, I went home to try and eat some dinner and get some sleep.

Bike racked and ready. She's a funny old bike, but fast enough for me.

Bike racked and ready. She’s a funny old bike, but fast enough for me.

"It's fun to stay at the....."

“It’s fun to stay at the…..”

Race Day

5.15am wake up, try to eat some breakfast. I managed a small coffee and a tiny bowl of Cheerios (quite plain, easy to eat). Amy picked me up to save me a mile walk. I made my way to my bike to put my spare inner tubes in the saddlebag and fill my bottles. Then to the beach. By 7.30 the nerves had vanished for some reason and replaced with a sense of calm. I had various comments from people floating round my head, one of which was from my wise Scottish friend @IronPugsley:  “Take the time to have a look around, take it all in and enjoy it.“. I did just that and looked at the faces of people. Most were looking nervous, some completely stone cold. A few smiling, so I exchanged smiles. Kept reminding myself I wanted to do this and it was an adventure. 7.45 and it was time for Amy to go into her holding area for her wave start. Good lucks exchanged and I’m on my own. I watched the pros start and then the next few waves. Then it was my turn. Wetsuit on, I got sort of in the middle of the pack and waited…

Swim

After an eternity of waiting the gun went off. At my point of the pack it wasn’t a run into the water, it was too busy for that, it was more like a crowd boarding a train, moving with conviction but without any pace. Found myself a space to swim and went for it. The water was opaque with kicked up silt and I could feel the bubbles of the person in front. 200m or so in, I’m not quite sure what happened but during a breath my goggles got removed from my face along with my cap. I stopped to find them a few yards back but whilst trying to put them back on someone swum over me and I took a lungful of sea water. Confidence knocked entirely I started to breaststroke my way after the pack which shot off into the distance. At the halfway marker I looked at my stopwatch which was showing about 19mins, so plenty of time left. It was very clear that I just needed to take it at my own pace. Pretty much on my own apart from a few others I alternated between 30secs of crawl and some breaststroke. I kept panicking with the crawl and I couldn’t find any rhythm. 45 minutes after the gun went,  I left the water feeling annoyed at myself but ready for the bike and pleased I made the cut off (which last year looked a bit tight).

Quite happy to be on dry land again

Quite happy to be on dry land again

 

Bike

OK... so how far is this ride again??

OK… so how far is this ride again??

Into T1, get bike gear on, grab bike and jog along the world’s longest transition (500m apparently). Mount the bike and off we go. Up into Alcudia old town where Matt Fisher blasted past me saying “come on Brian, mate!”. Sorry Matt, that was me actually pedalling quite hard. I literally have no idea how the fast bikers do it. It really is incredible. The pace I can maintain for a minute sprint, they’ll do for 90km. Amazing. The bike course is a single loop that follows the coast, takes in a 600 metre climb, then drops back down to undulating farmland for another 50km. I was starting to get into a zone on the bike, passing a few people that left the water ahead of me. Gentle cruise up the coastal road to Port de Pollenca gave me time to eat a 9Bar and have some fluid. Got to the first aid station at Pollenca having drunk most of my bottles so grabbed a water, an isotonic (yuck), and a half banana. The gentle climb from Pollenca up to the start of the main climb was only really noticeable by looking at the Grade reading on my Garmin.

Then passed a sign warning of a steep climb for the next 9km. Here we go, rattle through the gears one by one as speed drops and keep the legs spinning. The climb was never impossible, just long with very little rest. Although my bike is a heavy lump (it’s a pimped cyclocross bike) it does have the advantage of a triple chainring up front and a huge 32T gear on the back. I never used all the gears, but some people could have done with some extra options. At one point I was following a huge Russian guy called Ivan. He must have been about 6ft5, 18 stone and for some reason, on a small frame bike. He looked like  Donkey Kong and I was pretty sure he could just push his bike along with his hands. Unfortunately, he was getting slower and slower as the hill got steeper. I waited for a small gap in the passing bikes, clicked up a few gears and did a small sprint effort standing in the pedals. Other than that, the climb was just a case of looking at the reading from my Powertap and keeping it as smooth as possible between about 130W and 180W depending on traffic etc.

Bike seemed great on the climb.

Bike seemed great on the climb.

Once at the top of the climb (there’s a few false summits on the way!) I shifted into the big ring, zipped round the first corner and promptly found the chain had jammed in the front mech. Back pedalling made it worse so had to leap off the bike next to someone changing a puncture, pop the chain back on and away again. About 30 seconds wasted and more of an annoyance than anything else. Onto the descent proper and some of the riding standards were dreadful. I passed about 50-60 people on the switchbacks and didn’t get passed at all. There were some almost coming to a standstill, wobbling round the corner and pedalling like fury again for the downhills. Odd. Off the main descent, through Caimari village and hook a left onto the worst road surface you can imagine. Potholes, gravel, dust. Even cruising at 20kph felt too fast and the bike felt like it was being shaken to bits. Fortunately it didn’t last long until we got to section that was resurfaced on the Wednesday.

Along to Campanet  and the second aid station. The gap between aid station 1 and 2 is a whopping 40km and included the 600m climb. It really felt  like about 20km too far. They could have done with a water station (rather than a full aid station) at the top of the climb IMHO given it was starting to head towards 26-28degC by this point. Another banana, a Powerbar (yuck) and some more waters down me, the stretch from Campanet to the final aid station was only 15km along some half reasonable roads…. after seemingly barely enough time to finish what I was eating it was time to grab some more water and click off the final 25km back to Alcudia. Not sure if there was a tailwind or what, but down in the aero position along the flat smooth roads I was maintaining an easy 40-45kph with barely 130W of output. Lovely cruise and I’m feeling great. This’ll be over in no time.

Another cyclist pulls alongside me, shouting something at me in German that I don’t understand. Flustered, I stare at him as he’s looking back at me, about a metre in front and a metre to my left. I carry on at a cruise  as I’m aware there’s someone about draft legal distance behind me and I don’t want to slow them up too. I wait for him to pull the regulation 10m distance away. He has all the time in the world, this road is arrow straight, massively wide and empty infront of me. He turns away from me and all of a sudden moves over. Not a fast move, maybe two seconds or so. I’ve removed one hand from a tri bar moving it to the brake but it’s too late. His rear wheel just kisses my front and *BANG*. I’m down. 45kph according to the Garmin, and I have no bike underneath me.

Human beings weren't designed to decelerate like that on tarmac!

Human beings weren’t designed to decelerate like that on tarmac!

I don’t remember a vast amount from the tumble itself, it was probably too quick, but I do remember landing on my feet whilst watching my bike bounce through a field off the road. I then remember catching something out the corner of my eye and watching my two Powerbar water bottles come down from the sky like tiny yellow laser guided bombs onto the road, exploding in a shower of cold water. He didn’t stop. Nobody stopped. I’m standing up, shaking, blood dripping from all my fingers like a horror film poster.

I’m screaming at passing riders: “Someone call the next medic! Someone call the next medic!”. Silence. They all carried on. To the left and right, all I could see was heat haze from the tarmac, dusty fields to either side and a smattering of riders trying to ignore me. It reminded me of the film Rango where he falls from his tank onto the desert road.

I checked my limbs and everything seemed intact. Although incredibly painful I had movement in my arms. My left elbow though had swollen massively. It looked like someone had rammed something about the size,  shape and texture of a Mr Kipling cake slice under the skin. My legs seem sort of ok other than the skin. I walked over to the field to fetch my bike and get it back to the road. Lifting it was painful so it was more of a drag. It looked intact apart from the front tyre which was flat. Ah, that explained the enormous bang and hiss I heard as I was airborne.

It took ages to get my hands onto the zips of my saddlebag as they were shaking so much, but once I calmed down I managed to get the tyre levers and an inner tube out of my saddle bag. As I was fiddling with the levers trying to find a way to get the tyre off using my remaining working fingers (quite a lot of road rash) a course motorbike arrived. It was a Police rider with a bike mechanic on the back. The Policeman asked if I was ok while the mechanic fixed my front tyre in silent Spanish. Once the bike was found to be sort of working (all ok other than broken front derailleur which he managed to bodge into the middle ring) the Policeman ask if I needed a medic or if I was ok. I sort of took it as a good sign that they weren’t calling for an ambulance. I must have at least looked ok. I had a quick look at my Garmin which reckoned I’d travelled 70km, so I decided to try to ride the remaining 20km back to Alcudia. Worst case I could stop at the next point where there’s a course marshal. Beyond that, I could maybe ride to the next town. Maybe if I made that, I could just ride gently into Alcudia where the main event medical tent was. Yes, I’d do that. I’d ride to the med tent and let them sort me out.

Riding was ‘interesting’ to say the least. With my swollen (and very bloody) elbow I couldn’t put that one on the tribar armrest. The other hand had a whole bunch of missing skin from the palm so it was a weird riding technique with left hand grabbing the brake hood, right hand sort of holding the extension bar. Incredibly thirsty after standing in the sun and from the shock I went to grab a bottle, but remembered they’d both jettisoned in the crash. It was a long, hot ride back to Alcudia with zero fluid. Oddly though I still was passing people and maintaining a decent pace. Riding back to transition I started looking for the medical tent. I hopped off at the dismount line and ran (walked) over to a marshall to ask them. They said to ask the people in the tent at the exit of transition. So walk/jog the 500m to the other end and oddly felt fine.  OK, screw this, let’s at least start the run so I can say I tried even if I can’t finish.

Run

I grab my run bag, ditch my bloodied and shredded bike gear, put my run clothes on (apart from calf guards which look like they’d hurt my road rash) and trot out of the change tent. Ah, there’s the tent with the officials. I ask if I can just get a dressing/wash the blood off. They say I can do so in the medical tent at the finish line, but I need to go the other way to the 3 lap course to get there. She takes my race number and starts radioing through. I ask what she’s doing. She says I wanted to exit the race. No.. well.. I do. Desperately. But I really want to finish. I tell her I’ll run half a lap, if I struggle I’ll exit to the finish line area (and the medical tent) at that point. She radios something again which I hope is “Cancel that, the idiot is still running” rather than “Security, I need you to extract someone from the course”.  Off I go.

The run is a 3 lap course of the promenade, a couple of town streets and a main road out of town. It’s lined with spectators and feels like running in a carnival. I jog the 500m to the first aid station, down a ton of water/coke and whatever other fluid I can find. I last drank before the 3rd bike aid station at 65km and I’m parched. I take a gel on too and off I go to see how running feels.

A rare moment actually running...

A rare moment actually running…

It feels fine. Easy fine. I’m ticking along at sub 6min km’s and it feels effortless. In fact, more effortless than it normally feels. I check my heart rate and it’s 175. It can sometimes get above 180 on a run if I’m pushing but I feel like I’m putting in no effort, just gliding. I carry on for a few minutes longer and start to look around. People are really suffering in the heat. There are people dripping in cold water from sponges. There’s a couple of people lying on the floor. There’s one guy being sick. I normally suffer badly in the heat. Even lying around at home I feel odd in this kind of heat.  Hmmm. I take a walk break to cool off and gather myself. My heart rate doesn’t drop below 150 after a few minutes. A few minutes more walk and it’s still 150. Lots of people are pointing at me from the side of the road. Ah yeah, that’ll be the fact I’m dripping blood onto my white running top and down my leg. Sorry about that. At the pace I was going I must have looked a bit like a zombie.

Another aid station, another weird feeling run and I managed to let my sensible head take over. I was never going to set  a time to be proud of, I’d had quite a tumble, and potentially I wasn’t quite 100% myself. I realised the adrenaline could be masking all sorts of messages my body was trying to send me and my heart rate still wasn’t dropping below 150. I was probably very dehydrated from the bike, and potentially had some big injuries. I thought of Natalie and Ben, I realised I was potentially pushing myself into a unhealthy situation. I really shouldn’t do this run. I did a slow walk (always keep moving) and flipped through my Garmin to do some mental maths. Normally I find it easy to do that kind of thing in my head, but it was getting to the point where I had to ask another walking stranger for help. That was a pretty good sign all was not ok with me. As it transpired, at my quick walking pace (8:15/km) I was easily going to beat the cut off, so my run became a walk. A very social walk where I chatted to other non runners. An amusing walk where I was catching and passing people who were running/stopping. After a lap and a half I finally saw Nat and Ben waiting for me. I’d been slightly concerned that perhaps in the confusion they’d somehow been told that I was out of the race after a crash and were looking for me. I assured Nat I was fine, gave Ben a kiss and carried on my walk. An hour and a half more of chatting to strangers and it was time for the finish chute (where I ran, obviously).

This chap was Robbie, retired Fireman, 3 daughters. Life stories exchanged.

This chap was Robbie, retired Fireman, 3 daughters. Life stories exchanged.

Almost there!

Almost there!

Hurrah!

Hurrah!

Broken

Broken

Total time for the worlds slowest half marathon: 02:58:23

Final Thoughts

On one hand, I’m pleased I finished a 70.3. It’s something I thought I couldn’t do and really my aim was to just finish it. On the other hand, I’m gutted I made a bit of a hash job of it. I can’t really say I gave it my all in terms of physical performance. Yes, I ‘gave it spirit’ like an underdog high school team in an American movie, but that’s not what I’d trained for.

Swim wasn’t what I was hoping for at all, more to come there certainly and it’s all mental rather than physical.

Bike, looking at the Garmin data, removing the 10 min crash stop would bring the bike split to 3:30 putting me about 20th from last in my age group. I’d probably have ridden a harder last 20km sans crash so would have gained a few more places. In hindsight I reckon I totally undercooked the bike. I could have upped my target watts and still felt fine. I think I was a bit cautious there.

Run, no idea. I’ve not run in so long,  but I reckon with a bit more walking than running I’d get an easy 2:30 run, putting me again about 20th from last.

On the plus side, I finished the race feeling fine. Grabbed my medal and made my way to the med tent where I saw 6 bodies with IV fluids, 2 with oxygen masks. Screw that. I prefer my way. Death or glory isn’t really my style. Once they had room for me, a very pretty Spanish girl cleaned me up with iodine (YOUCH), popped a few stitches in my elbow and sent me on my way. Nat picked me up, took me home and helped me clean myself up. I hugged my family and nearly burst into tears. It could have gone horribly wrong. It didn’t, but it could have. Later in the evening I met up with Amy, Roland and James for a well deserved pizza, a couple of beers and a wonderfully girly cocktail.  I did the uncool thing and wore my medal to dinner.No sleep at all that night as just couldn’t get comfortable and my head was full of “what if”s”

The next few days were understandably slower than usual but managed to meet up with Pete and his lovely wife who were out for the weekend to recce the course and watch the race. Would I do it again? Yes, in a heartbeat. I think I’m going to enter for the 2015 race as a warm up for a full distance IM later in the year. It’s an easy location, stunning course, relatively sensible weather and a good atmosphere.  Anyone else fancy it? I promise the 2015 report will be snappier……

Thanks

Thanks to Natalie. Thank you for putting up with my never ending stream of stupid ideas that demand your energy as well as mine. Dragging yourself to Mallorca at 30 weeks pregnant with a 2.5yr old and a relatively useless husband is the work of an angel.

Thanks to Ben. Sorry buddy, I know you hate it when you watch me race and I always seem to run away from you. You’re always in my thoughts and one day we’ll do this silly kind of nonsense together ok ?

Best support crew EVER.

Best support crew EVER.

Thanks to Amy for your help during the week. Having someone around who knows what they’re doing helped settle my nerves loads. You’re a superstar and it was a total pleasure. Your race result was fantastic and well deserved (Amy finished in 5:32:35, some 2hrs 10 mins quicker than me!). Dead proud of you.

Thanks to all those who supported me on Twitter/Facebook before, during and after the race. I could genuinely recall the lovely things you all said the day leading up to the race and it really did help me get my head straight. It’s funny the voices that go through your head. Sometimes it’s people you barely know because they’ve said just the right thing.

Finally, thank you for reading.

Links

Ironman Split Times

Garmin Ride Data

Garmin Run Data

Complexity, Requirements and The Perfect Cup Of Tea

The words any software developer dreads hearing are “Can you just change this one thing for this one user/client/company”. The requests usually have perfectly valid reasons, but it’s sometimes hard to explain to the person asking for the feature why it’s more work than just the code. Most of the time it’s not the coding that’s terribly hard, it’s making it just so for that particular user, testing it, maintaining it and then you have the nightmare two years down the line when you have to change code that touches it.

Much has been written about system complexity but it tends to read like this:

complexity1

Great for your PhD but terrible as a demonstration/case study.

Hopefully, the following little story can serve as a way of visualising the pitfalls when a system ends up in an ever increasing cycle of complexity.

In The Beginning: A Village Fete Tea Stand

complexity2

Agnes, Beatrice, Clive and Doris decide to run a small tea stand at the village fete in their home village of Systemton in the Yorkshire Dales. They do a quick run to Costco for a giant box of Yorkshire Tea and some huge cartons of milk, Agnes borrows a tea urn from the village hall, Clive pops into IKEA for 50 cheap tea cups and a handful of teapots and they’re all set. They decide on a nice round 20p per cup to make charging customers very simple. Doris will brew the tea, Beatrice will take the cash, Clive will collect the used cups and Agnes will wash them up. They knock up a quick sign that says “Tea, 20p” and hang it outside.

Very soon, they’re running flat out and things aren’t running smoothly. To cut down on the wait they’re pre making the tea, but it’s hard to keep track of how long each pot has been brewing. Some customers are complaining it’s stewed and some are having lightly coloured water. Also, some customers are complaining it’s too milky. It seems that milk is being added to make the colour the same, but that means lightly brewed tea is only getting a splash, whereas over brewed tea is getting far too much milk.

Being an ex RAF engineer, Clive decides it’s time for a system. He labels the teapots A, B, C, D and E and knocks up a chart. Using his watch he carefully notes the time each pot is filled and removes the teabags after exactly 5 minutes. They can adapt to demand by having between 1 and 5 pots on the go at once meaning the tea stays hot. He also finds a measuring cup that looks about right and milk is poured into that first before going into the cup, so each cup has exactly the same amount.

After 30 minutes or so Agnes overhears some friends say it was a nice refreshing cuppa, but perhaps a bit on the strong side. She feeds this back to Doris who adjusts the brewing time down to 4 minutes. Feedback seems to be much better from everyone and they settle on 4 minutes as the ideal brew time. They now have a system for the perfect cup of Yorkshire, word gets around about how good the tea is, and they collect 20p per cup from happy customers all day long. In fact most customers not only have 20p ready, most of them pop it in the box themselves making life very easy indeed for Beatrice so she can help Doris with the tea making.

Complexity-chart1

Lessons

  • Take what you do and systemize it
  • Refine the system with feedback and improve your product

Year 2: More Options

It’s fete time again, and this year Beatrice accompanies Clive on the Costco run. As well as the many jugs of semi-skimmed and the box of Yorkshire Tea bags, Beatrice puts a box of her preferred PG Tips in the trolley. “Good idea” says Clive.

Fete day comes, the stall is running like clockwork with Doris using Clive’s system from last year. She also had a brainwave and brought along 5 kitchen timers and she sets them off when each pot is filled. When the timer beeps, she takes the teabag out of the relevant pot and bingo, the perfect cup of tea.

Except there’s a snag. They have both boxes of tea on display behind the stall, and both are being offered to customers. Doris tries to adapt to the new option by having pots of Yorkshire and a pot of PG Tips on the go, she just makes pots A, B, C and D Yorkshire and E is brewed with the less popular PG Tips. Everything goes fine until Beatrice has a cup of tea herself and chooses her favourite PG. Yuck.. over brewed. She feeds back to Doris that PG Tips only takes 3 minutes to brew. Doris adjusts her countdown timer for pot E to 3 minutes and they now have perfect cups of tea in two flavours. 20p’s are collected all afternoon and word gets out about the perfect cuppas.

Late in the afternoon after most locals had left, a coach stops off at the fete. As they’re paying their entrance fee, Agnes has the idea to get all the teapots brewing in preparation. Four pots of Yorkshire and a pot of PG on the go. The first customers appear and order PG. As do the other 50 from the coach. Oh dear. This coach was from Lancashire and they don’t want the Yorkshire. Doris has to suspend brewing of the 4 pots of Yorkshire throw them away and restart with PG. Halfway through she remembers that the 4 timers were set for 4 mins not 3. Hastily she changes the process so they don’t stew.

Still, crisis largely averted, the coach load pay their 20p’s, our four friends collect the money and sit down for a well earned rest!

Complexity-chart2

Lessons

  • Extra requirements can sometimes require unique, or altered processes further downstream.
  • Even slightly more complexity makes it harder to react to small changes in the marketplace.
  • Choice can work against you. If only one tea was on offer they’d likely have still have sold as many cups.

Year 3: Demand Increases and Scaling is Required.

The four friends dust off their IKEA mugs and teapots, do a Costco run again, pick up the milk, the Yorkshire and the PG

Fete day arrives and they’re blessed with glorious weather, so much so that the village is overrun with visitors from the surrounding area eager to see the ever expanding Systemton Fete. Agnes, Beatrice and Clive are rushed off their feet collecting/washing cups to re-use whilst Doris makes the tea.  The system of multiple teapots works well in general and they sort of adapt by having a split of 4-1 or 3-2 depending on demand, not forgetting to change the timers each time of course!

Lunchtime comes and the queue is out the door, there’s simply not enough cups or teapots to scale to the required throughput. Agnes asks her friend Edith to rustle up a load more cups from peoples houses as well as two more teapots. Brewing speeds up again and the queue decreases. However, not all is rosy. Customers from years before have come to expect the perfect cup of tea and it seems some are getting overly milky or stewed tea again. What on earth is going on? They have a quick internal management meeting and the problem is obvious. In their attempt to scale to demand they introduced different sized teapots and cups. The 4min/3min times and the measure of milk no longer work. The system is broken. Their only options now are:

  1. Scale back down, make customers wait for the perfect cup.
  2. Supply poor quality tea to a percentage of customers, but quickly.
  3. Try to modify the process with some different variables for the different sized pots and cups which will require extra man power.

Clive stops collecting cups takes one of the IKEA teapots and the two ‘new’ ones (therefore massively reducing productions whilst they’re offline). He measures the volume of each and he decides that because they’re 50% and 150% the size of the IKEA ones, brew times should be:

  Yorkshire PG Tips
Ikea Pot 4 mins 3 mins
Large Pot 6 mins 4.5 mins
Small Pot 2 mins 1.5 mins

He does a quick test with each pot using PG (as it takes less time than to test Yorkshire) and it all seems ok.

He jots the times down and hands it to Doris who rolls her eyes at him. She asks about the milk measure and the different sized cups. They look around at their customers… there’s the 50 identical cups and 50 more borrowed cups each of which a completely different size to one another. Whilst possible, the work involved to measure and test each cup would be horrendous, not to mention from an operational perspective doing different measures. Doris has more than enough to do without different milk measures for 51 different types of cup. If they choose to go down this path they’ll need a dedicated person just for milk measures (who, incidentally, won’t be needed when it gets quieter and they can drop back to the 50 original cups).

Milk measures aside, they’re now running at a huge scale, supplying at least twice as much tea as before. They’ve had to rope in Edith to help Doris of course as managing the various combinations of tea type/pot size is unmanageable for one person. In fact, it’s getting too much for the pair of them when demand changes and they need to swap to say 3 PG Tips pots, but Beatrice can help out as she doesn’t need to be looking after the payment full time as it’s simply 20p per customer and 9 out of 10 customers put a coin in the box themselves anyway.

Complexity-chart3

Lessons

  • Scaling can itself add more variables and complexity
  • Scaling is hard when the system is already complex
  • Adding more options when the system has already been scaled is just as hard

Year 4: Requirements By Committee and Homemade Cake

This year, Mavis, the head of the village committee decides she like the tea stand and it should have her input. She elects herself into the team as Tea Stand CEO, and decides that they need to be offering a few more exotic teas such as Darjeeling, Assam and Jasmine. And biscuits.

The news is broken to the rest of the group on the Friday before the fete. “Oh dear”, thinks Clive as off he goes in search of tea.  He can get PG and Yorkshire (and milk and biscuits) from Costco, but the others require a search. Eventually he finds the others but at a much much higher cost. “Hmmm, not sure Mavis thought about the cost of these fancy teabags, but we’ll have to work something out”

He gets home and calls the others. In preparation for tomorrow they decide not to get caught out this time and they test the new teas beforehand and they work out what tastes right. All testing is carried out with the standard IKEA pot.

Tea Brew Time
Yorkshire 4 mins
PG 3 mins
Darjeeling 2 mins
Assam 3.5 mins
Jasmine 5 mins

Great. Except they’ve not had any budget to increase their capacity of IKEA teapots (though fortunately, they did manage to buy 100 more cups so at least those are now standard) so there’s still this issue of the two other pots. Using the same method as last time they do some guestimates and hope the values are right.

  Yorkshire PG Tips Darjeeling Assam Jasmine
Ikea Pot 4 mins 3 mins 2 mins 3.5 mins 5 mins
Large Pot 6 mins 4.5 mins 3 mins 5.25 mins 7.5 mins
Small Pot 2 mins 1.5 mins 1 min 1.75 mins 2.5 mins

This chart is written out, and they then turn their discussion to pricing. Working on the cost per teabag it’s decided that Yorkshire and PG are 20p/cup and the other 3 should be 30p/cup as 20p is just too low to make any money on the more expensive exotics. Of course, this is assuming the customer wants milk. 99% of PG/Yorkshire customers have milk, it’s pretty optional with Assam, but a no-no with Jasmine and Darjeeling. It’s decided that it’s unfair to charge all customers for milk, so actually the milk should be charged at 2p/measure, so the prices are adjusted to 18p and 28p with a 2p extra if you have milk.

Simple.

Of course there’s also the biscuits which are charged at an easy 10p each regardless of biscuit (some will end up paying over the odds for very small biscuits from the selection box, but that’s life)

They throw away their sign from year 1 and make a new one.

complexity3

Fete day arrives again, and it’s quickly apparent there’s a problem. Customers are asking lots of questions before placing the order. Some are unsure about the different tastes of the fancy teas. With Doris and Edith brewing, Beatrice on the till, Clive collecting, Agnes washing and Mavis doing…. CEO things, there’s nobody to answer customer queries, so they enlist the help of Florence who is given some training in Tea and put at the front of the stall to field questions.

Within half an hour, brewing operations has fallen apart as well. Beatrice is too busy now with the more complicated money side of things and the serving of biscuits to help out like last year, leaving Doris and Edith not only having to deal with the Yorkshire/PG in different sized pots problem, but they have the other teas. Problem is, those teas aren’t being ordered in sufficient quantity to justify a pot on the go so they’re being made in the cups. Reacting to the change and altering the process by brewing in cups means of course the well previously researched pot times are all wrong for the special teas, so they have to rapidly re-test them in cups while in full production. It also means someone has to monitor the times of each cup. Doris and Edith decide they’ll look after the pots and they draft in Edith’s husband Graham to do the individual teas using the old style chart and watch method of timing the brew according to the pre-set brew times (there’s nowhere near enough kitchen timers). The gang muddle through, each working flat out to keep the production line running.

At lunchtime, Mavis starts chatting to her friend Olive who is running a home made cake stall. Olive lets on that she’s having a slow day. Ever one to help out a friend and maybe do some business along the way, Mavis has an idea. Why not re-sell Olive’s cakes on the tea stand. They’ll take the money and give Olive a 30% cut. Everyone’s a winner baby.

Mavis brings over a tray of each of Olive’s 20 different cakes complete with the price label on each cake, along with some plates and cutlery and places them on the stand. Unfortunately, this just serves to confuse customers more with all this choice and they soon turn to Florence to ask about the cakes as well as the tea. Florence though, being diabetic knows nothing about the cakes as it’s just not her area of expertise, she’s now a tea maestro though. She answers as best she can until Mavis grabs Olive’s daughter Heather to answer cake queries.

The end of the day arrives and our exhausted team take the proceeds, calculate Olives cut by using the sales ledger, split the remaining proceeds 9 ways between them and go home exhausted.

Complexity4a

Lessons

  • Ever increasing complexity at the core of the business has now required dedicated sales people, increased the work in acquiring the raw materials, increased the work in billing, marketing and stock control. These are processes both upstream and downstream of the main brewing core of the business.
  • If you keep expanding your options, at some point, your process will have to split and you have multiple processes with different teams. You can only scale vertically so far.
  • Specialist products require specialist knowledge, and when it comes to scaling, you have to build each team rather than just the one. The overheads are now vastly higher than in year one.
  • System integration often requires more reporting than you’d imagine to make it work for both parties, without detailed reports of cake sales, Olive won’t be able to run her cake production optimally.
  • Can you imagine scaling this to deal with double the customers?

Year 5: The Final Chapter, The Team Go To India!

Word has finally got out about the original perfect cup of Yorkshire from year one, the product that got them their reputation. The team is invited to the World Tea Championships high up in the Indian Himalaya. The team do their shopping, buy some more standard teapots (no messing about this time) and pack their gear. The week before, they get together all 9 of them and revisit all the brewing times of the various teas, all in pots this time. After a lot of focus grouping amongst external friends and past customers they arrive at some very specific, perfect times, slightly improving on the original Yorkshire and the latter PG tips times as well as the fancy teas.

  Yorkshire PG Tips Darjeeling Assam Jasmine
Ikea Pot 4.1 mins 3.2 mins 2.4 mins 3.1 mins 4.3 mins

Arriving in the Indian Himalaya the intrepid tea-makers trek high up into the hills to the venue for the competition.  They now have a finely honed system, perfect timings, standardised cups/teapots, a dedicated workforce, many options of tea, Fox’s biscuits and Olive’s home made cakes. There’s no way they can lose this.

The big day arrives and the scene is hectic. There’s thousands of tea lovers from all over the world and our team get brewing. The first few customers don’t seem massively impressed. Still, undeterred they continue serving from the various pots, keen to bring Yorkshire and PG Tips to the world. Except all the customers are hating it. Could it be they’re not used to it? Our team try a cup of their finest Yorkshire….. and it’s terrible. Something has broken and the brew is off. Way off. The altitude of the venue is lowering the boiling point of water to 85degC, not even close to what they’re used to.

Our crew can either try to re-engineer their processes by trialling new brew times for each of the 5 teas, buying more pots (since brewing is longer, they’ll need more pots on the go, and the extended brew time makes it even harder to react to demand as the lead time is longer) and maybe train yet another staff member in brewing to keep up with capacity now that brewing is the bottleneck.

-OR-

They simplify.

They ditch everything apart from their beloved Yorkshire, they trial a new brew time at the lower boiling point. They use ALL their teapots for Yorkshire, they ditch the separate charge for milk and put the cup price back to 20p. They average out the price on the cakes to another 20p to simplify billing. They give away biscuits to customers who bring friends along rather than trying to charge and complicate the pricing structure. All day long, they serve the perfect of tea to hundreds of happy customers, not one of which asks them if they can have Jasmine.

Complexity4

Lessons

  • A complex system makes a change in a key ingredient, or a change in market or environment a potential disaster.
  • Staying simple doesn’t mean not evolving. You can add more products, just do it in a streamlined way than minimizes branches in your processes.
  • A simplified process makes it much easier to adapt and scale. All hands moved to brewing and with almost all the team on the case they coped with demand.
  • Remember your core business.
  • Doing a simple thing very well pleases more people than lots of things done just above average.
  • This super efficient tea business can now be replicated to any country and with just one test brew to check for altitude and water changes the whole thing can be copied with minimal training, low overheads per outlet and a simple method of quality control. A Perfect Cup Of Yorkshire(tm) will be next to every Starbucks in the world.

 

Conclusion

Adding increasing numbers of options, features, widgets and conditions leads to an ever increasing burden of complexity not just in building a system, but in designing, testing, supporting and don’t forget operationally. It’s not always the best experience for the customer either.

Our team didn’t quite end up where they started though. They added biscuits and delicious homemade cakes. They also productionized the tea making and refined the brew time. They also learnt a lot along the way.

Now, time to put the kettle on…

Mini mouse review – Microsoft Sidewinder X8

My old stalwart Logitech cordless mouse has been slowly getting worse over the last few months. It was struggling with the wood grain on my desk, and even on a better surface the pointer would occasionally jump.

Looking around at mice I wanted something with super high DPI for accuracy, a couple of buttons for doing ‘other things’ and it needed to be cordless. Getting the wire wrapped around your keyboard is so 1990s

Seemingly, the problem with wireless mice is a vast majority now seem to be aimed at laptop users, so there’s no charge cradle, they just run on an AA battery or two and have an off switch. Great until you leave it on overnight and have a flat mouse in the morning.

On the DPI front, most regular mice seem to top out at 2000DPI or so, but who do we know that always need vastly over the top kit? Yes… gamers!

After a hasty 10 minutes research I located this puppy on Amazon:

41GKGnzX3fL._SL500_AA300_[1] 

http://www.amazon.co.uk/dp/B001DCELH2

 

Yes, it looks ridiculous. The odd styling touches, bizarre logos and occasional lights make it the Citroen DS3 Racing of the mouse world,  but for someone looking for a responsive wireless mouse it’s perfect.

Have used it for a week now and the accuracy over the old Logitech is very useful. Also useful are the adjustable DPI buttons on the top, so I can have Fast/Normal/Slow But Accurate settings direct from the mouse. Designed for gamers doing stupid things in first person shooters, but incredibly useful to be able to slow it right down for masking fiddly details in photo software.

Oh, and the bit I like the most though I’ve not needed it yet, there’s no cradle. Instead, there’s a magnetic puck on the end of a thin cable so you can charge it whilst you use it like a corded mouse. My old Logitech required you to pop it in the cradle and twiddle your thumbs for a few hours if the battery ran out.

All in all, a top product at a reasonable price. I just wish it didn’t look quite so Halfords.

Managing clients patched calls with a single click

The status quo of call patching

With our pureJAM service you can have calls patched to you. You simply set your call instructions for one of 4 statuses. You can change this status via Twitter/SMS/Web page so you have pretty good flexibility about how we handle your calls. You can have us try to patch to your mobile, your landline, both, or neither for each status. See the below example for what a call instruction looks like when being set up.

1

In the above example, when the operator takes the call for me and I’m ‘Busy’, they’ll politely put the caller on hold, call me on 02072070007 and see if I want to take the call. They’ll tell me who they have on the line, and if I want to take the call they’ll put them through, if I don’t they’ll explain to the caller that I’m unavailable and take a message.

A better solution

Wouldn’t it be great though, if you could have this ‘discussion’ with our operators much much more quickly and less intrusively. A lot of us spend our time behind a computer, but that doesn’t always mean you’re available to take calls. Maybe you’re on another call, maybe you’re on a video chat or maybe you just want to get some work done.

So, I’ve developed a very prototype system for doing just that. It uses a desktop client that sits in your taskbar (and allows you to change your status, gives you indication of unread messages etc). You must be a pureJAM client (otherwise there’s no point as you can’t have calls patched!), have a PC, and you must have the .NET Framework 3.5 or later. It’s a Windows ClickOnce application so it’s easily installed from a URL. It also connects to our systems on port 80 and uses standard HTTP so it’ll work behind any firewall/NAT. If you want the download link then you’ll need to register for the Beta program (details at the end of the post).

Step 1: Instruction Setup

Once you’ve installed the desktop client and logged in and out of your online portal you’ll see the desktop client logs in as well (using a rather neat way of linking a browser to a Windows application). Then, if you go into your contact instructions you’ll see a new menu option under ‘Patch To’:

2

You then get a choice of action if we don’t hear back from you i.e. you’re not at your computer (incidentally, you can install copies of this desktop client on multiple computers and it’ll send the request to any that are logged in)

3

And this is what the status summary now looks like:

4

Step 2: A caller comes through to us

When our operator gets a call for you, they’ll get the basic details from the caller, and then they’ll click the (See If They Want This Call) button.

5

Step 3: We let you know about the call

Within a few milliseconds, a packet of data is sent to your PC (via our really cool comms system that you don’t need to care about) containing the basic call information that our operator got from the caller. You’ll get a notification bleep and you’ll then have a choice of what to do with the call.

6

Step 4: Our operator gets your response

Using the long polling techniques I developed in Project Totem, your response is instantly  pushed backed to the operators screen and they can handle the call how you want them to.

7

 

Beta testers required

If you’re a pureJAM client and want to test this system out you’ll need to be a PC user (no Macs I’m afraid) and you’ll need the .NET Framework 3.5 or later. Most Vista/Windows7 PCs should work fine. XP will work fine provided it’s been kept up to date ! To register for the Beta program, please login to your pureJAM account and send a U2U to your Account Manager entitled something like “Desktop Client Beta”.

viewmessages.com Architecture

OK, so we’re not the biggest site in the world but we have a fair amount of data, a fair amount of users and speed is very important to me so it’s important everything is as fast as possible. A few people have asked what our architecture is and I thought it’d make an interesting post. As is always the way with these things it’s easier to describe with a diagram:

PearlServerArchitecture

Web Servers

content.viewmessages.com

First of all, we serve images/bulky javascript and CSS from Amazon Cloudfront CDN which is an incredibly cheap way of offloading those things to the Amazon infrastructure. It also makes the platform much much snapper for our American users. If you even have a basic website it’s worth looking into using Cloudfront if only because it gives you a second domain to pull you data from which allows the browser to parallelise more downloads.

totem.viewmessages.com

Totem is my own long polling server I developed to allow instant communication to the users browser. This allows things like instant new message notification. In short, your browser uses JQuery to request a script from Totem. If there’s no new messages, Totem will sit there for 40 seconds and return nothing. Your browser will then re-request the script and wait for another 40 seconds. If you get sent a U2U for example 5 seconds into the 40 seconds, the web server/background server dealing with the U2U sends a notification to Totem which creates a bit of Javascript to display the U2U notification and sends it back as the response to the original request that was made 5 seconds previously. For more on Totem, read my Project Totem blog post.

static.viewmessages.com

Because we use a web cluster to serve the main HTML we need a central server for avatars and other central data that we don’t push to Cloudfront. The only challenge here was getting content to it. Security in IIS from the main webcluster meant I couldn’t access the machine directly to I had to use a SQL database as a proxy

http://www.viewmessages.com

The main web serving is done by a cluster of IIS machines. These are cheap commodity machines in the Google style. 2GB Ram/2GHz Dual Core CPU/80GB drive. Nothing fancy or expensive. By using multiple cheap machines instead of one big expensive one we get vastly better availability (they can be brought offline for updating), far better performance (if you add up the total computing power) at less cost. It’s a win-win other than it makes the software development slightly more complex at times.

Each machine runs a copy of SQL Express to write access logs to (Which are then copied to the main SQL box when things are quiet) and to store a whole bunch of reasonably static information (such as configuration) to reduce the load on the main SQL box. Each machine can do front end web serving, back end task processing or both. As we need more capacity we can simply add more machines. The load balancers will send the users request to a particular web server using a session cookie. If the server goes down, the failover happens within 10 seconds and you’ll be transparently placed onto a different server. 

The back end task processing is something I’m particularly pleased with as it allows the processing load to be distributed across as many machines as we need. At the moment these are the same machines that serve the front end web stuff but at a later date will be split off into a dedicated back end cluster. All the back end processing is done by requesting webpages from a queue. If you want to read about how we process background tasks heres my blog post about it

Background Servers

 Background / Offline Processing

As mentioned above, this is done using queues of webpages and is processed by the main web cluster

Main SQL Store

Nothing interesting here really I’m afraid. Just a reasonably beefy Dell machine with data replicated to a hotspare backup.

Solr Server

I’m now using Solr to generate the data for the new Message Analytics Feature . I’ll do a blog post about it at some point in the future but it’s incredibly fast compared to using XML data with SQL. Doing a ‘Group By’ on an XML value in SQL was taking around 1200ms for a particular data set (with an unloaded server). Using Solr on a *much* less powerful machine took 20ms. It’s an incredible piece of software if slightly tricky to use.

Memcached

The staple of every high performance website. Memcached is a memory based data store. I don’t use it to store reasonably static data as that’s done in the ASP.Net cache object (which is 10x quicker due to it being on the machine itself), but I use Memcached to store precompiled data that’s used across machines. For example, if you get sent a U2U it’s a background task that ‘delivers’ it to your inbox. This task puts the message in your inbox, adds it to the search database, then takes the most recent 10 U2U’s for you and recompiles the HTML you see in your ‘Recent U2U Messages’ widget on your homepage and inserts it into Memcached.  The background task then notifies the Totem server about the U2U, Totem notifies your browser, your browser requests the new HTML blob back from the webserver and guess what? It’s already been generated and the webserver just grabs it from Memcached. The beauty of using it over the ASP.Net cache is that cached objects can be shared across machines.

Memcached is a great bit of software and we’ve had absolutely zero issues with it.  The current stats from our memcached instance are below:

STAT uptime 28625365 (nearly a year)
STAT time 1280509032
STAT pointer_size 32
STAT curr_items 30626   (It’s 100,000 or so during busy periods)
STAT total_items 10108777
STAT bytes 9450225
STAT curr_connections 17
STAT total_connections 10040
STAT connection_structures 24
STAT cmd_get 39701711
STAT cmd_set 10108777
STAT get_hits 33158267  (It’s saved a LOT of SQL reads!)
STAT get_misses 6543444
STAT bytes_read 3126086257
STAT bytes_written 821258193  (It’s served 800GB!)
STAT limit_maxbytes 524288000

And to think, I was almost tempted to use Velocity instead. You can read why I didn’t.

Summary

By applying a bit of thought and leveraging the right technology for each part of the puzzle we’ve got a platform that *way* outperforms a traditional single big webserver setup. We also have minimal load on the main SQL box by using quite aggressive caching (In memory on the local webserver, in Memcached and in SQL Express on the local webserver).

Hot air extraction – more efficient server room cooling

 

In addition to the power for servers, a huge cost we have is cooling them. 6kW of servers is going to require some chilling. Our data room has air conditioning and it works very hard for a living particularly in summer where the heat differential on the aircon exchanger outside is lower. In a big data centre you’d pump chilled air into a ‘cold aisle’ in front of a load of racks, and then have a ‘hot aisle’ behind them where you suck the air back into the A/C. Unfortunately our building wasn’t designed with this in mind so we simply have a wall mounted unit that cools the whole room. The problem with cooling the room though, is there’s no way of making sure the servers see chilled air, they might get air that has come directly from the back of the rack and sucked back round again.

Whilst doing some tidying up I spotted one of our old extraction fans from years gone by. When we were a much smaller company, air was drawn into the room at one end and extracted at the other end. It kept things cool enough until we started to need more equipment and then A/C was the only option.

Anyway, below, you can see the unused fan and our main server rack to the left.

freshaircooling1

There are probably some very expensive hot air extraction systems on the market, but I figured there was no point in spending a lot of cash to trial it out. B&Q to the rescue for some gaffa tape and guttering pipe. Add in the old box from my Herman Miller ‘Mirra’ chair, and an hour of creativity and we have a working hot air extraction system…..

freshaircooling2

I simply made a baffle infront of the fan and added ducting that goes down behind the server. It’s not pretty, but it does work:

freshaircoolinggraph

There… proof it works! We dropped the temp measured at the top of the rack by a degree. Air intake temps on the servers lowered even more. Our SQL server was drawing in 24 degree air previously, and is now a lot more chilled. (21 degrees!). The UPS unit on the floor beside the rack had a similar drop from 25deg to 22deg.

We’ve massively reduced the strain on our air con unit for the grand sum of about £50 and the overhead of a 100W fan. (which is more than offset by the potential savings in air con for that room)

The next thing to try is adding curtains from the side of the rack to the wall to force the hot air into the extracted area.