TIL: How to use jq to extract new posts from a JSON Feed

Thu, 02 Feb 2023 00:00:00 +0000

I’m currently looking into ways to automate some stuff around new posts on this page (be it blog or TIL post) directly during the page build on GitHub Actions. For this, I first need to be able to reliably detect new posts, from a bash run step. So here’s how to do that with jq.

The idea is to get the current feed.json prior to publishing the page, and then compare it to the one that was just generated during the build. If there are any differences, we know that there are new posts and can trigger further actions from there.

# Get current feed.json
curl -s https://foosel.net/til/feed.json > feed.current.json

# Get new feed.json
cp public/til/feed.json feed.next.json

# Compare the two, this is where the magic happens
jq --slurpfile current til.current.json --slurpfile next til.next.json -n '$next[0].items - $current[0].items' > til.json

Let’s go through this jq command there:

--slurpfile reads in the given files and makes it accessible as an array contained in the given variable. In this case we read in til.current.json and make it accessible as $current, and also read in til.next.json and make it accessible as $next.
-n doesn’t wait for input on stdin.
'$next[0].items - $current[0].items' subtracts the items from the new feed from the items in the current feed¹.
> til.json writes the output to til.json.

til.json will then contain all new items (as long as there weren’t more than the feed’s item size), can be uploaded as an artifact and then used in further jobs².

The indexing (e.g. $new[0]) here is needed due to --slurpfile creating an array from the read file. I admittedly need to experiment more with this option to fully understand it, but for the purpose here it works. ↩︎
My current goal is to move my announcements on Mastodon for new posts from my NodeRED install got the page build, and also send any webmentions for links contained in new posts as well. ↩︎

TIL: How to add JSON Feed support to Hugo

Sun, 29 Jan 2023 00:00:00 +0000

In order to add JSON Feed 1.1 support to Hugo you need to first add a new jsonfeed output format in config.yaml:

mediaTypes:
  application/feed+json:
    suffixes:
      - json

outputFormats:
  jsonfeed:
    mediaType: application/feed+json
    baseName: feed
    rel: alternate
    isPlainText: true

This adds a new media type application/feed+json with the extension json and creates a new output format jsonfeed rendering into that media type with a base name of feed (so feed.json as recommended by the JSON Feed spec).

This then needs to be added to the outputs it should be generated for - on this page I’ve only added it to sections. Again, in config.yaml:

outputs:
  home:
    - HTML
    - JSON
  section:
    - HTML
    - RSS
    - jsonfeed

Finally, a template needs to be created so that Hugo can actually render something. I’ve put this into layouts/_default/list.jsonfeed.json (following the expected naming scheme of list..):

{{- $pctx := . -}}
{{- if .IsHome -}}{{ $pctx = site }}{{- end -}}
{{- $pages := slice -}}
{{- if or $.IsHome $.IsSection -}}
{{- $pages = $pctx.RegularPages -}}
{{- else -}}
{{- $pages = $pctx.Pages -}}
{{- end -}}
{{- $limit := site.Config.Services.RSS.Limit -}}
{{- if ge $limit 1 -}}
{{- $pages = $pages | first $limit -}}
{{- end -}}
{{- $title := "" }}
{{- if eq .Title .Site.Title }}
{{- $title = .Site.Title }}
{{- else }}
{{- with .Title }}
{{- $title = print . " on "}}
{{- end }}
{{- $title = print $title .Site.Title }}
{{- end }}
{
    "version": "https://jsonfeed.org/version/1.1",
    "title": {{ $title | jsonify }},
    "home_page_url": {{ .Permalink | jsonify }},
    {{- with  .OutputFormats.Get "jsonfeed" }}
    "feed_url": {{ .Permalink | jsonify  }},
    {{- end }}
    {{- if (or .Site.Params.author .Site.Params.author_url) }}
    "authors": [{
      {{- if .Site.Params.author }}
        "name": {{ .Site.Params.author | jsonify }},
      {{- end }}
      {{- if .Site.Params.author_url }}
        "url": {{ .Site.Params.author_url | jsonify }}
      {{- end }}
    }],
    {{- end }}
    {{- if $pages }}
    "items": [
        {{- range $index, $element := $pages }}
        {{- with $element }}
        {{- if $index }},{{end}} {
            "title": {{ .Title | jsonify }},
            "id": {{ .Permalink | jsonify }},
            "url": {{ .Permalink | jsonify }},
            {{- if .Site.Params.showFullTextinJSONFeed }}
            "summary": {{ with .Description }}{{ . | jsonify }}{{ else }}{{ .Summary | jsonify }}{{ end -}},
            "content_html": {{ .Content | jsonify }},
            {{- else }}
            "content_text": {{ with .Description }}{{ . | jsonify }}{{ else }}{{ .Summary | jsonify }}{{ end -}},
            {{- end }}
            {{- if .Params.cover.image }}
            {{- $cover := (.Resources.ByType "image").GetMatch (printf "*%s*" (.Params.cover.image)) }}
            {{- if $cover }}
            "image": {{ (path.Join .RelPermalink $cover) | absURL | jsonify }},
            {{- end }}
            {{- end }}
            "date_published": {{ .Date.Format "2006-01-02T15:04:05Z07:00" | jsonify }}
        }
        {{- end }}
        {{- end }}
    ]
    {{ end }}
}

By default, this generates a feed with summaries only. If you want a full content feed, set params.showFullTextinJSONFeed to true in config.yaml.

The relevant docs for custom media types, output formats and template locations can be found here.

On the Papermod theme the above will automatically cause something like

<link rel="alternate" type="application/feed+json" href="https://foosel.net/til/feed.json">

to be added to the head of the page, as needed for discovery. In other themes you might have to do it yourself.

The result of all of this is something like this.

jsonfeed on foosel.net

TIL: How to use jq to extract new posts from a JSON Feed

TIL: How to add JSON Feed support to Hugo