Rowan’s Blog | Rowan Manning

Operational excellence: one year on

2022-11-04T11:22:00+00:00

A year ago, Nayana wrote a blog post outlining some changes to the way we support FT.com. As well as giving an excellent overview of what our “OpsCop” team did, it set out some changes that we intended to make. We wanted to stop being reactive and start proactively finding ways to enable operational excellence on FT.com.

I’ve been working on the new team for about a year and I think the change in focus has been a huge success. We are now a team of three permanent senior engineers, a technical product manager, and a delivery manager. We recently renamed ourselves to the Reliability Team to better reflect the long-term vision of the team, our expertise, and our changing role in the organisation.

How the team works

We still triage issues

We’re still the team responsible for triaging issues that come into FT.com, however, this is now only a small part of what we do. Nayana said in her original post:

“Our vision is to get to a place where the ft.com team in partnership with the stream-aligned product teams and the platform team spend 90% of their time improving the overall reliability aspects of ft.com and only spend 10% of their time on triaging issues that the group has never seen before.”

We’ve met Nayana’s original target, dropping from ~90% to ~10% of our time spent on triage. Lower-severity issues now get sent directly to the owner of the system without ever reaching our team, this gives us space to focus on more critical underlying issues.

There’s definitely less toil than there used to be, and we’re focused more on actionable outcomes of an incident. A recent example is the FT.com site navigation going down and managing to take out several applications. We identified the root cause as well as some other improvements we could make to increase the resilience of this service. We spent a week mobbing with the team who own this system, increased knowledge of how it fits into our other systems, and fixed some fairly critical issues. Previously we would not have had the capacity to make those underlying changes, especially in such a short period of time.

Another thing that we’re now doing with our issue triage is trying to spot patterns over time. For example, in early 2022 we were seeing quite a few issues related to our membership services. On the customer-facing website, these are presented as error pages, and our customer care team would often bring these issues directly to us. Rather than continue to react to these issues when they came up, we built some tools to help us quickly identify and route these issues to the team responsible for billing and subscribers.

We proactively seek out potential issues

A key point made in Nayana’s original post was around being proactive rather than reactive.

“We believe that this would help reduce the reactive operational work that teams have to do giving us more time as a group to focus on the proactive, preventive and predictive maintenance of ft.com systems and services.”

This is something that we’ve made a clear focus over the last year, as the reactive approach was not scalable: more FT subscribers lead to more issues and support requests, which then require more engineering effort.

We all had some assumptions about which parts of the FT.com stack were less-than-reliable because we’ve always had relatively good monitoring. What we didn’t know was the why; whether there were any common issues we see across all of our systems.

We did a lot of investigating: picking apart our error logs to look for patterns, digging into the code for different systems to find common issues, reviewing the root causes of incidents to see if there are any shortfalls in the way we design software. We did some deep dives into one or two of our more problematic systems and wrote down everything we found. This put us in a good position to make recommendations to our teams, and we’ve been rolling out some changes based on this research.

We own code and systems

We gave ourselves a few months’ grace period for the team to bed in, but then we started taking ownership of all the libraries and systems related to the reliability and stability of FT.com. This gave us the ability to make improvements without relying on other teams and also took the pressure off some of the product teams who just don’t have as much time to focus on this area.

We’re still slowly ramping up but we’re now at 16 repos, all of which we understand well enough to reliably support.

As well as taking ownership of existing code, we’ve also been looking at how we can produce new software to improve FT.com’s reliability. After identifying some areas we could clearly improve, we released Reliability Kit, a suite of Node.js libraries and guides that will help us standardise errors and logging. We’re slowly migrating our applications to this. It’s an area we previously under-invested in and we’re already seeing some big improvements to the observability and reliability of the systems which have adopted it.

An example of the improved error logging we’ve rolled out with Reliability Kit

Are we solving the problems of one year ago?

Nayana highlighted several key issues in her original blog post.

No metrics and Service Level Agreements for incoming issues

We’re on our way to solving this; we’re definitely focused on reducing recurring issues and we have a better process for fixing incoming issues. We haven’t quite got there with SLAs (service level agreements) but some work has been done on adding SLOs (service level objectives) for systems.

Some large operational issues without a clear owner to drive strategic changes

As discussed already, we’re now investigating the broader operational issues that impact FT.com and owning the work to address them. We’re able to focus on some large technical challenges that previously wouldn’t have been possible.

No systematic process for creating a shared sense of ownership across ft.com

We don’t yet have a systematic process, but we’re going to investigate supporting engineers seconding into our team. We’re also very open and visible in the work we’re doing — we’re sharing regular internal blog posts, design docs, and the results of our investigations.

What are the new problems?

A year is a long time in software development, and we’re always identifying new problems to solve. These will help us decide what the team should be doing going forward.

We could support our customer care team far better than we do

Our customer care colleagues aren’t being set up for success when it comes to supporting our customers. They’re given little direction from FT.com engineering on how our systems fit together and we could be doing a far better job at sending them to the right team.

We’re looking at how we can give them the tools they need for this, firstly with some better error signposting on FT.com. This will reduce the time between a user calling customer care and getting a fix rolled out, and it’ll further reduce our time spent on triage.

There’s not a lot of shared ownership of site performance

We’re investigating how we can improve performance across the whole of FT.com. We’re currently not doing badly, but we don’t have all the tools in place to ensure that we don’t take steps back in terms of web performance. This could easily become a future issue.

We’re focused a lot on the website

Customer Products is the group that looks after both FT.com and our mobile apps. As you can probably tell from reading this post, there’s been more of a focus on the website from our team for now. Having a shared centralised team to focus on the reliability of the website and the apps is something we’d like to explore.

Here’s to another year

We’re excited to have made it to a full year, and we’re really proud of some of our achievements. I’m sure there will be a lot more to come over the next year.

Thanks for reading

Halving my GitHub Pages Build Time

2022-08-21T10:35:00+01:00

Recently I spent some time overhauling the build process for this website because it was really slow. After nearly 100 weeknotes (including a lot of images to resize) my build/deploy process was taking 6–7 minutes. Locally my entire site builds in 500–800ms so I decided to investigate where my performance issues were.

The TL;DR here is that my builds have dropped to ~3 minutes, more than half my original times. I’ve also removed some uncertainty about exactly when my site will get deployed. Here are the steps I went through:

I realised that I wasn’t caching Hugo’s build or resources/_gen folders in CI, which meant that the entire site had to be built every time. No wonder this was slow! Caching these folders based on a hash of the config, content, data, and themes folder reduced build times by a lot and I’m now taking full advantage of how fast Hugo is. Here’s the step I defined:

 - name: Cache built website
   uses: actions/cache@v3
   env:
     cache-name: cache-website-v1
   with:
     path: |
       build
       resources/_gen       
     key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('config', 'content', 'data', 'themes') }}
     restore-keys: |
       ${{ runner.os }}-build-${{ env.cache-name }}-
       ${{ runner.os }}-build-
       ${{ runner.os }}-

I wasn’t taking advantage of GitHub Actions’ ability to run jobs in parallel at all, I was waiting for the linting to pass before running tests, and then waiting for the tests to pass before building the site.

The linter, tests, and build have absolutely no shared tasks except that they all have the need to use Node.js and npm. I moved this to a new install job and made the linting, testing, and build happen in parallel. The final deploy job relies on all three of these passing. This gave me so much more speed.
One large bottleneck was the way I was deploying. Until very recently the only way to build GitHub pages with Hugo was to push to a separate branch or folder then a separate GitHub action would do the deploy under the hood. So, no matter how fast my own workflow was, I was still having to wait for a second one to trigger.

Very recently GitHub launched a beta for deploying to GitHub Pages from your own GitHub Action. This is game-changing for me because I have full control over the zipped artefact that gets deployed. No more waiting for GitHub to decide when to deploy my pages, I can trigger it myself immediately after the build 🎉

The GitHub Actions starter workflows repo contains examples of how to do this with many other static site generators.

The new steps used are:

These three steps had a huge impact, reducing my build and deploy times by more than half. Some of the issues were me writing suboptimal actions originally, but it’s also been massively helped by the new GitHub features.

You can view an example run on my website repo here and you can see my full GitHub Actions workflow here.

Webmentions for your Static Site

2021-07-06T22:45:00+01:00

Some time ago, my friend and colleague chee told me about Webmentions. I’ll be honest, I didn’t fully understand what they were. It took me about a month to actually look into them, and wow it’s been a while since I’ve been this excited about a technology!

I’m very late to the party, which started back in 2012, but now I’ve arrived I’m ready to fully commit (whatever the web dev equivalent of drinking Kalimotxo Blondes and dancing to Old Town Road is). After reading this post, I’m hoping that you will:

Be at least half as excited about Webmentions as I am,
Have some idea how to implement Webmentions on a static website,
Try sending a Webmention to this page just to see your comments appear below.

This post assumes some knowledge of web technologies and that you’re able to publish your own website.

What are Webmentions?

For the uninitiated, I should probably attempt to explain what Webmentions are. If you know this part then feel free to skip ahead.

Webmention is a web standard for mentions, conversations, and other social interactions (such as likes and reposts). It defines a way for these interactions to happen across different websites without the need for a central server, allowing for decentralized interactions with other people on the web. At its core, a Webmention is a notification from one site to another that says “Hi, somebody linked to your website from this one”.

Webmention Flow

A typical Webmention flow might look like this, for example:

Kara writes an interesting post on her website (which is set up to receive Webmentions).
Simon reads Kara’s post and links to it in a new post on his own site.
Simon’s publishing software sees this link and sends a Webmention to Kara’s website. This Webmention lets Kara know that her interesting post has been linked to, including the URL of Simon’s post.
Kara’s website checks that Simon’s post actually contains a link to her post, and then it includes this information on her site.

The end result of this flow is that Simon’s post links to Kara’s, and Kara’s now links back to Simon’s. The information can be displayed in any way and, in this scenario, Kara might want to display Simon’s post as a comment on her post. They have achieved this and avoided many of the issues with centralized commenting systems:

The commentary on Kara’s post is decentralized – it is stored on Simon and Kara’s websites, rather than relying on a third-party server. Both people own all the content they’ve written.
Neither person has had to hand their personal information over to a third party.
Kara does not need to maintain a “comments” form or any kind of sign in for people who want to comment on her posts.

So that’s webmentions! If you need more of a deep dive than this, then I recommend starting with this article.

Static Site Difficulties

In my previous example, I talked about servers being able to send and receive webmentions. With a static website (one that is plain HTML or is built from a Static Site Generator), there is no server involved. This means that both sending and receiving webmentions require a few more hoops to jump through:

Receiving webmentions isn’t possible without some kind of server, you need a URL that can receive source and target URLs and store the result somewhere and this isn’t possible with plain old HTML.
Sending webmentions manually is possible, but it adds a new step for you to complete every time you publish content. It’s possible to fully automate sending during the build step of your static site.
Displaying webmentions requires you to store them somewhere, which means you need a static site generator that supports rendering content from data files (e.g. JSON), or you need to accept that you can only render webmentions with JavaScript on the client.

Receiving Webmentions

As mentioned above, you need some kind of server to receive webmentions. Luckily there are a few services which can do this for you. I opted for Webmention.io which is a free hosted service. To set up the basics of receiving webmentions, you’ll need to:

Set up IndieAuth for your website:
- Make sure a social profile of yours (e.g. Twitter, GitHub) links to the home page of your website. You can do this via the edit profile page of these services (Twitter’s is here and GitHub’s is here).
- Make sure that your home page includes a link to the same social profile page with a rel="me" attribute. I have these links in my site header, but they could go anywhere:
```
<a href="https://twitter.com/rowanmanning" rel="me">
    @rowanmanning on Twitter
a>
```
- Because both the user profile page on the third party website and your website now include rel="me" links pointing to each other, IndieAuth can verify that you are who you say you are. The standard behind this is called RelMeAuth.
Sign into Webmention.io using your newly configured IndieAuth website. You’ll need to authenticate via one of the third parties you linked to.
Add a element to the of every page on your website where you want to receive webmentions – I just added it to every page. This link acts as an instruction to other Webmention-enabled sites that you’re using Webmention.io and that they should send webmentions here:
```
<link
    rel="webmention"
    href="https://webmention.io/YOURDOMAIN.COM/webmention"
/>
```

Now you’re ready to receive webmentions! If somebody who publishes webmentions links to your website, it will appear on your Webmention.io dashboard for you to see.

An additional step for me was that I wanted to capture interactions on social websites like Twitter. The large social sites don’t implement Webmention themselves, however, another third party service can be used to watch for social interactions on other sites and publish webmentions on their behalf.

Setting up Bridgy was relatively quick, and it can connect to many social sites and forward on any interactions with your content as webmentions. I added Twitter, which means that any time someone tweets a link to my content, or likes/retweets links to my content, I’m notified and can display it on my website.

The same two likes displayed on Twitter (left) and my website (right)

Storing Webmentions

So now you’re receiving webmentions, it’s time to store them alongside your website code so that we can move onto the next step, displaying them. For this, you’ll need for your website to include a build step, which is already needed for many Static Site generators.

My website is built with Hugo and is hosted on GitHub Pages, so I already have a build step that uses GitHub Actions to automatically generate the static HTML and deploy it via the build branch.

Hugo allows for your website to access JSON files in the data folder so, for me, it makes sense for my build step to fetch webmentions from Webmention.io via their API and store them in this folder for use later.

I won’t document the entire script that I wrote, but I went with the approach of fetching new webmentions every 30 minutes using GitHub Actions scheduling. You can find documentation on scheduled actions here.

My script started out pretty much like this (you can get your own Webmention.io API key from your settings page):

const got = require('got');

const response = await got('https://webmention.io/api/mentions.jf2', {
    responseType: 'json',
    searchParams: {
        domain: 'rowanmanning.com',
        'per-page': 1000,
        token: process.env.WEBMENTION_API_KEY
    }
});

console.log('Webmentions!', response.body.children);

It got complicated fairly quickly. The things I added were:

Storing the last time we fetched webmentions to reduce unnecessary work,
Converting the webmention URL to a safe filename, e.g. /weeknotes/36/ to /data/webmentions/weeknotes--36.json,
Saving each of the webmentions for a specific URL into the relevant JSON file,
Committing these new files and pushing them to the repo so that we don’t need to fetch webmentions on every build.

You can look at the full script here, and you can also see an example of these saved JSON files on GitHub: here’s the JSON file for my Yeehaw blog post.

So now that we’re storing our webmentions alongside our website content, the next step is displaying them on each of the pages.

Displaying Webmentions

With webmentions saved as JSON, the next thing we need is to display them alongside the content they link to. This step will differ depending on which static site generator you’re using. For my site, built with Hugo, it meant adding something like this to my page templates:

{{/* Replace "/" in the relative permalink of the post with "--" */}}
{{/* this matches the file names of our JSON files */}}
{{
  $slug := .RelPermalink
    | replaceRE "^/|/$" ""
    | replaceRE "/" "--"
}}

{{/* Access Site.Data.webmentions[slug] */}}
{{/* which is the JSON file matching our page */}}
{{/* default to an empty array if no mentions exist */}}
{{
  $webmentions := index $.Site.Data.webmentions $slug
    | default slice
}}

{{/* ...loop over webmentions and display them */}}
<ul>
  {{ range $webmentions }}
    <li>
      Mentioned on
      <a href="{{ .source }}">{{ .source }}a>:
      {{ .content | safeHTML }}
    li>
  {{ end }}
ul>

Webmention as a standard doesn’t specify how you should display webmentions on your website. It also doesn’t advise on how to safely include other peoples’ content on your website. The above code by itself is an attack vector because it will inject any HTML sent as part of a webmention. This could include malicious