Since I have some images in the docs that I want to be able to reference without having to use relative URLs (../../../../images/), especially since that would tie things in OctoPrint’s source tree structure too close to things in its documentation tree structure that might or might not end up being in a different repository in the future, I needed a way to use absolute URLs here (/images/). But since the docs will most likely also end up being hosted on a version specific subpath of docs.octoprint.org, just using (host) absolute URLs would not work either and break.

There’s several issues on the MkDocs issue tracker about workflow problems caused by this, but the suggested workarounds like using macros to prefix a variable’s contents to related URLs didn’t work for me due to me also heavily relying on mkdocstrings, and anything contained in docstrings is not processed by macros¹.

So I got the idea to implement a minimal MkDocs plugin that would just turn all URLs contained in href and src attributes that are prefixed with a custom schema site: schema into site relative URLs, with this effect. Example:

Using a hook I could register a callback for the on_page_content event that would then replace all URLs as needed in the generated page HTML.

And this is the resulting site_urls.py:

import logging
import urllib.parse
import re

import mkdocs.plugins

log = logging.getLogger("mkdocs")

SITE_URLS_REGEX = re.compile(r'(href|src)="site:([^"]+)"', re.IGNORECASE)

@mkdocs.plugins.event_priority(50)
def on_page_content(html, page, config, files):
    site_url = config["site_url"]
    path = urllib.parse.urlparse(site_url).path

    if not path:
        path = "/"
    if not path.endswith("/"):
        path += "/"

    def _replace(match):
        param = match.group(1)
        url = match.group(2)
        if url.startswith("/"):
            url = url[1:]

        log.info(f"Replacing site:{match.group(2)} with {path}{url}...")
        return f'{param}="{path}{url}"'

    return SITE_URLS_REGEX.sub(_replace, html)

that I’ve registered as a hook in my mkdocs.yaml like this:

hooks:
  - site_urls.py

Seems to work just fine, both for images and links! 😄

Update 2023-07-28: I’ve now published this as a proper plugin on PyPI, see mkdocs-site-urls. With that, all you need to do - given you are already on MkDocs 1.5 or newer - is installing the plugin via pip install mkdocs-site-urls and then adding this to your mkdocs.yaml:

plugins:
  - site-urls

The format of the lines in octoprint.log is a simple %(asctime)s - %(name)s - %(levelname)s - %(message)s, so a log with an error and attached exception looks like this:

2023-01-30 17:50:45,704 - octoprint.events.fire - DEBUG - Firing event: Disconnecting (Payload: None)
2023-01-30 17:50:45,704 - octoprint.events - DEBUG - Sending action to <bound method PrinterStateConnection._onEvent of <octoprint.server.util.sockjs.PrinterStateConnection object at 0x000001635CB6EE50>>
2023-01-30 17:50:45,705 - octoprint.plugin - DEBUG - Calling on_event on action_command_notification
2023-01-30 17:50:45,705 - octoprint.plugin - DEBUG - Calling on_event on action_command_prompt
2023-01-30 17:50:45,705 - octoprint.plugin - DEBUG - Calling on_event on announcements
2023-01-30 17:50:45,705 - octoprint.plugin - DEBUG - Calling on_event on file_check
2023-01-30 17:50:45,706 - octoprint.plugin - DEBUG - Calling on_event on firmware_check
2023-01-30 17:50:45,706 - octoprint.plugin - DEBUG - Calling on_event on pluginmanager
2023-01-30 17:50:45,706 - octoprint.plugin - DEBUG - Calling on_event on softwareupdate
2023-01-30 17:50:45,706 - octoprint.plugin - DEBUG - Calling on_event on tracking
2023-01-30 17:50:45,711 - octoprint.plugin - DEBUG - Calling on_event on mqtt
2023-01-30 17:50:45,732 - octoprint.events.fire - DEBUG - Firing event: Disconnected (Payload: None)
2023-01-30 17:50:45,735 - octoprint.events - DEBUG - Sending action to <function Server.run.<locals>.<lambda> at 0x000001635BDB2CA0>
2023-01-30 17:50:45,750 - octoprint.events - ERROR - Got an exception while sending event Disconnected (Payload: None) to <function Server.run.<locals>.<lambda> at 0x000001635BDB2CA0>
Traceback (most recent call last):
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\events.py", line 197, in _work
    listener(event, payload)
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\server\__init__.py", line 1212, in <lambda>
    octoprint.events.Events.DISCONNECTED, lambda e, p: run_autorefresh()
                                                       ^^^^^^^^^^^^^^^^^
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\server\__init__.py", line 1195, in run_autorefresh
    autorefresh.stop()
    ^^^^^^^^^^^^^^^^
AttributeError: 'RepeatedTimer' object has no attribute 'stop'
2023-01-30 17:50:45,753 - octoprint.events - DEBUG - Sending action to <bound method PrinterStateConnection._onEvent of <octoprint.server.util.sockjs.PrinterStateConnection object at 0x000001635CB6EE50>>
2023-01-30 17:50:45,755 - octoprint.plugin - DEBUG - Calling on_event on action_command_notification
2023-01-30 17:50:45,756 - octoprint.server.util.sockjs - DEBUG - Socket message held back until permissions cleared, added to backlog: plugin
2023-01-30 17:50:45,758 - octoprint.plugins.action_command_notification - INFO - Notifications cleared
2023-01-30 17:50:45,758 - octoprint.plugin - DEBUG - Calling on_event on action_command_prompt
2023-01-30 17:50:45,758 - octoprint.plugin - DEBUG - Calling on_event on announcements
2023-01-30 17:50:45,759 - octoprint.plugin - DEBUG - Calling on_event on file_check
2023-01-30 17:50:45,759 - octoprint.plugin - DEBUG - Calling on_event on firmware_check
2023-01-30 17:50:45,759 - octoprint.server.util.sockjs - DEBUG - Socket message held back until permissions cleared, added to backlog: plugin
2023-01-30 17:50:45,763 - octoprint.plugin - DEBUG - Calling on_event on pluginmanager
2023-01-30 17:50:45,764 - octoprint.plugin - DEBUG - Calling on_event on softwareupdate
2023-01-30 17:50:45,764 - octoprint.plugin - DEBUG - Calling on_event on tracking

What I now wanted is for grep to spit out just the ERROR line and the attached stack trace:

2023-01-30 17:50:45,750 - octoprint.events - ERROR - Got an exception while sending event Disconnected (Payload: None) to <function Server.run.<locals>.<lambda> at 0x000001635BDB2CA0>
Traceback (most recent call last):
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\events.py", line 197, in _work
    listener(event, payload)
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\server\__init__.py", line 1212, in <lambda>
    octoprint.events.Events.DISCONNECTED, lambda e, p: run_autorefresh()
                                                       ^^^^^^^^^^^^^^^^^
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\server\__init__.py", line 1195, in run_autorefresh
    autorefresh.stop()
    ^^^^^^^^^^^^^^^^
AttributeError: 'RepeatedTimer' object has no attribute 'stop'

For this I needed a way to set grep to match multiple lines and do a (non-matching) look ahead for the end. It turns out that the secret to success here is to treat the whole input as one line, use Perl compatible regex mode, and make sure to set the multiline flag. After some fiddling around on regex101.com and reading up on Perl’s regex options¹, I came up with the following:

grep -Pazo '(?m)^\N+\- ERROR \-\N*\n(^\N*?\n)*?(?=\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \- )' octoprint.log

Let’s walk through this:

• -P enables Perl compatible regex mode
• -a enables text mode
• -z turns all newlines into null bytes and thus treats the whole input as a single line for finding matches²
• -o only outputs the matched part of the line (otherwise we’d get the whole file printed out)
• (?m) enables multiline mode
• ^\N+\- ERROR \-\N*\n matches the first line of the error, which is the one that starts with the timestamp and package and contains the word ERROR
• (^\N*?\n)*? non-greedily matches all following lines of the error, which are anything but a newline followed by a newline
• (?=\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \- ) is a positive look-ahead that matches a line starting with a timestamp again, which signifies the end of the error’s lines

Hooray, it works 🥳:

❯ grep -Pazo '(?m)^\N+\- ERROR \-\N*\n(^\N*?\n)*?(?=\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \- )' octoprint.log
2023-01-30 17:50:45,750 - octoprint.events - ERROR - Got an exception while sending event Disconnected (Payload: None) to <function Server.run.<locals>.<lambda> at 0x000001635BDB2CA0>
Traceback (most recent call last):
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\events.py", line 197, in _work
    listener(event, payload)
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\server\__init__.py", line 1212, in <lambda>
    octoprint.events.Events.DISCONNECTED, lambda e, p: run_autorefresh()
                                                       ^^^^^^^^^^^^^^^^^
  File "C:\Devel\OctoPrint\OctoPrint\src\octoprint\server\__init__.py", line 1195, in run_autorefresh
    autorefresh.stop()
    ^^^^^^^^^^^^^^^^
AttributeError: 'RepeatedTimer' object has no attribute 'stop'

(And yes, I’ve fixed the error that lead to this stack trace as well 😉)

The original version of this post was published as a Twitter thread on March 23rd 2020. I figured I should give it a more permanent home here since IMHO it was a quite fun story.

Since everyone can use some entertainment right now, how about a battle story on how a year ago I spent almost two weeks trying to wrap my head around a really weird issue of a lagging GCODE viewer and overall print progress reporting in OctoPrint and finally figuring it out?

Our story begins around the release of 1.4.0, when a new topic on the community forum showed up:

Curious issue with print progress

The print progress figures on my Octopi setup are lagging behind the actual print. […] Nothing is broken - anything I throw at it (an Ender 3) prints fine but as a print progresses, the percentage complete, current layer, and sync’d gcode viewer gradually lag behind what is actually being printed. For example, on a print with 400 layers, as the last layer is printed the reported progress and current layer is around 96% and 385 respectively. If I do a quick calculation of the displayed Printed/Total file size figures it works out at 96% but what it has actually printed is over 99%. When the print finishes the numbers jump to 100% and 400 and everything is fine.

[…]

This was indeed a very curious issue, since due to the nature of the communication with the printer and buffering in the firmware the progress is usually rather slightly ahead than behind. Some quick testing on my end showed no reproduction, however more and more people chimed in with the same observation.

I was stumped.

My first approach was to collect information from those affected by it. Printer model, firmware version, installed plugins, used slicer and so on. It soon turned out that all affected installations were using Ultimaker Cura as the slicer.

A quick test by the OP with a different slicer confirmed that it indeed just occurred with GCODE sliced by Cura for him, same file in another slicer had everything work as designed. However, comparing the GCODE revealed no immediate differences that would explain this, and what actually is in the file also doesn’t really play into progress tracking. My own experiments with Cura failed to reproduce.

Convinced that the issue must be some sort of delay between the backend and the frontend – maybe due to network issues? – I whipped up a plugin (since deleted) to log progress on both ends to a log which could then be shared and analysed. The first results came in an guess what? I had barked up the wrong tree, the reported progress was identical. So back to square one.

I still couldn’t reproduce it on my end and was starting to get really angry at this issue 😅 I finally threw a copy of some GCODE files now shared by the reporter of the issue on my own printer and finally I could reproduce. Which doesn’t mean I had any idea WTF was going on though.

After many test prints, head scratching and going through the files with a comb I finally noticed something. The files with the issue had CRLF (or \r\n) line endings. Those without (including my own sliced files) had just LF (or \n) line endings.

So that made me go 🤨 Some cursing and breakpoint setting later I had proof that the reported progress in backend and frontend was flawed to begin with. I could see that a line was being reported with a file position that it actually was not located at in the file, and which instead belonged to a couple lines earlier. Which meant my positions were reported wrong right at the source – with a lag. And then it suddenly hit me.

But before I can tell you what was happening I need to give you some background on how OctoPrint reads GCODE files it’s printing in order to understand what was going on. Printed files are read line by line because that is how they are sent to the printer. For that OctoPrint uses the readline method of the file stream. And that works by reading chunks of data from the file until a line separator is found, returning everything read up to this separator and saving the rest for the next line to be read. That means the file will have to be read further than what is returned. And that means that the position in the open file as reported by tell on the file stream will always be slightly ahead. For progress reporting in OctoPrint however I need to know the exact byte position of each line in the file. So what I do instead of relying on the internal and slightly ahead file position is that I increase my own position indicator by the length of the line read from the file. And this is where my problem was located.

It turns out that for some reason I wasn’t getting the lines back from readline with the original line endings attached. Instead I always got LF, even for files with CRLF. And that means I was counting one byte short for every single line in CRLF terminated files. One byte short per line doesn’t sound like much, but that adds up through a file with several hundred thousands of lines, to a point where progress reporting will be off by whole layers the further in the print and thus the file you are.

But what was the reason for this popping up in 1.4.0? I hadn’t modified the code in question at all. It had been the same since 2016 actually. Well, it turns out that a tiny change during the Python 3 compatibility migration done to a helper function I used in that code had interesting side effects: switching from codecs.open to io.open.

It turns out that io.open (and thus Python 3’s built-in open) by default will open text files in “universal newlines mode” (see PEP278), meaning it will happily parse every common line ending, but convert it to LF before returning. Which caused my off-by-one issue in files with CRLF.

And the fix? Setting newline="" on the open call:

diff --git a/src/octoprint/util/comm.py b/src/octoprint/util/comm.py
index 67191a7af..a6dfc1e24 100644
--- a/src/octoprint/util/comm.py
+++ b/src/octoprint/util/comm.py
@@ -4078,7 +4078,7 @@ def start(self):
 		"""
 		PrintingFileInformation.start(self)
 		with self._handle_mutex:
-			self._handle = bom_aware_open(self._filename, encoding="utf-8", errors="replace")
+			self._handle = bom_aware_open(self._filename, encoding="utf-8", errors="replace", newline="")
 			self._pos = self._handle.tell()
 			if self._handle.encoding.endswith("-sig"):
 				# Apparently we found an utf-8 bom in the file.

The moral of the story? Don’t trust your file position calculations. I could have saved myself a lot of time on debugging this if I had just looked there first instead of assuming this code to be fine 😅

In the end, even a year later, I still have no idea why Cura produced CRLF code for some and LF for me, but I also never really looked hard. A UNIX vs Windows issue can be ruled out here since the affected parties and me were all using Windows. It made me learn something about io.open and was a valuable lesson on wrong assumptions however!