What It B-Log

I found it oddly difficult to get my Plex Media Server logs on my MacMini M2 sent to a remote syslog server. This post is how I finally got it working.

Papertrail syslog server

I use Papertrail for my remote syslog server which provides a syslog endpoint that looks like logs#.papertrailapp.com:#####. They have pretty solid documentation on setting up new "Systems" (as they call them) as sources of syslogs and how to configure them to send their logs to your Papertrail destination.

No ARM builds of remote_syslog2

Papertrail's suggestion for "app log files" on macOS is remote_syslog2. However there is no Darwin ARM build in their releases. I may have been able to use Rosetta but I didn't try. I reached out to Papertrail support to ask if there was a supported version for Apple silicon Macs and was eventually directed to ask via a Github issue on the Github repo which I never got around to.

What about the built-in syslogd?

I never figured out how to configure syslogd to send an arbitrary log file to a remote destination. It was easy to configure macOS to send its system logs to the remote destination (and in fact, Papertrail's "Add System" page tells you exactly how). But selecting a specific file and routing its contents to a remote syslog server I never got working. Maybe I missed something.

Blah blah blah, what did you use?

I ended up using rsyslog which proved to be its own brand of frustrating. The documentation is mediocre (or I'm dumb, which is a real possibility worth keeping in mind) and there's two different configuration formats that can coexist in the same file, one newer than the other. Their documentation has the following to say about about the two configuration formats.

basic - previously known as the sysklogd format, this is the format best used to express basic things, such as where the statement fits on a single line. It stems back to the original syslog.conf format, in use now for several decades.

The most common use case is matching on facility/severity and writing matching messages to a log file.

advanced - previously known as the RainerScript format, this format was first available in rsyslog v6 and is the current, best and most precise format for non-trivial use cases where more than one line is needed.

Two supported formats means the internet is full of a mixture of the two when looking for solutions. There also doesn't seem to be complete feature parity between the two so and ended up using both while trying to enable debug logging.

Dude, get to the fucking point (aka tl;dr)

Anyway, I finally got it working with the following steps.

1. Install rsyslog via homebrew: brew install rsyslog
2. Start it as a service via homebrew: brew services start rsyslog
3. Modify the config file at /opt/homebrew/etc/rsyslog.conf (default brew installed location)

My config:

module(load="imfile")

global(
  workDirectory="/opt/homebrew/var/run"
)

# Forward logs to remote syslog server
ruleset(name="papertrail") {
  action(Type="omfwd"
         Target="<subdomain>.papertrailapp.com"  # Replace with your Papertrail endpoint
         Port="<port>"                           # Replace with your Papertrail port number
  )
}

# Monitor Plex Media Server log file
input(Type="imfile"
      File="/Users/<user>/Library/Logs/Plex Media Server/Plex Media Server.log"
      Tag="PlexMediaServer" # Set syslog Program Name
      Severity="info"
      Facility="local6"
      RuleSet="papertrail"
)

There are two key pieces in the above configuration, the modules omfwd and imfile.

We use the input module imfile to ingest a file as an input and give it a Ruleset of papertrail. Above that, we have defined a ruleset using the output module omfwd to send things matching the ruleset to our Target.

Now we need to restart rsyslog with brew services restart rsyslog. At this point you should be able to see your Plex Media Server logs show up on your remote syslog server.

Troubleshooting

If you're not seeing any logs on your remote location, first make sure Plex Media Server (PMS) is generating logs where you expect. You can tail the file in your File parameter of the input directive. If you're worried about PMS not generating logs, you can try turning on debug logging in your server by following the steps in the Plex Media Server support docs.

If Plex is generating logs at the expected location, you can turn on debug logging for rsyslog and see if it provides any clues. First, open your config file (/opt/homebrew/etc/rsyslog.conf) and add $DebugLevel 2

Example:

module(load="imfile")

$DebugLevel 2 # <== add outside any other directives

global(
  workDirectory="/opt/homebrew/var/run"
)

# Forward logs to remote syslog server
ruleset(name="papertrail") {
  action(Type="omfwd"
         Target="<subdomain>.papertrailapp.com"  # Replace with your Papertrail endpoint
         Port="<port>"                           # Replace with your Papertrail port number
  )
}

# Monitor Plex Media Server log file
input(Type="imfile"
      File="/Users/<user>/Library/Logs/Plex Media Server/Plex Media Server.log"
      Tag="PlexMediaServer" # Set syslog Program Name
      Severity="info"
      Facility="local6"
      RuleSet="papertrail"
)

Then take a look at the rsyslog logs. When installed with homebrew, the default log location is: /opt/homebrew/var/log/rsyslogd.log.

Primarily I followed these two guides but there were some missing pieces I had to figure out myself.

Put colima start CLI options in a config file

In the override.yaml file you can put things like cpus: 8, memory: "8GiB", etc so that you can simply run colima start. An example ~/.colima/_lima/_config/override.yaml is below.

vmType: "vz"
cpus: 8
memory: "8GiB"
mounts:
  - location: "/host/filesystem/location1"
    # by default, the mountPoint will be the same as the location, meaing
    # /host/filesystem/location1 will be mounted in the Docker VM as the same path
    writable: true
  - location: "/host/filesystem/location2"
    writable: true

More example configuration and details are in the Lima repository.

Giving Colima access to host filesystem locations

The above config file also points out another missing piece, how to give the Docker VM access to your host filesystem mounts.

You can also do this during colima start by using the --mount /host/location:/docker/vm/location:w where :w makes the directory writeable.

Basically everything on the left side of a services dir:dir entry in your docker-compose.yml's volumes: sections should be mounted for Colima. However, you don't have to mount each individual subdirectory. For instance, you can mount everything in a Nas network folder by just doing /Volumes/YourSharedFolderName and your containers should have access to /Volumes/YourSharedFolderName/SubDir as well.

Start Colima on system boot

If you, like me, want Colima (and your containers) to start on boot, you can easily do so by running brew services start colima