Key Takeaways
- Linux system logging changed with the introduction of systemd, which centralizes and manages system, boot, and kernel log files in a binary format.
- The journalctl command is used to read and filter system log messages, allowing users to steer and seek through logs.
- Users can customize the display format of journalctl output, such as using different output formats appreciate short, json, verbose, or cat, to focus on specific log details.
Linux system logging changed with the introduction of systemd. Learn how to use the journalctl command to read and filter system log messages.
Centralized Logging
No stranger to controversy, the systemd system and service manager introduced a significant change in the way system logs are gathered. Logs used to be located at different places in the file system according to the service or daemon that was creating them. But they all had one thing in common. They were plain text files.
With systemd all the system, boot, and kernel log files are collected and managed by a central, dedicated logging solution. The format they are stored in is a binary one. One thing this facilitates is being able to extract the data in different formats, such as JSON, as we shall see. It can also make it easier to cross-reference related information that would have previously been recorded in separate log files. Because the data is now held in a single journal, the data from several sources of interest can be selected and displayed in a single interwoven list of entries.
journalctl is the tool used to work with the journal.
journalctl With No Frills
You can invoke journalctl with no command line parameters:
journalctl
journalctl displays the entire journal, with the oldest entries at the top of the list. The list is displayed in less, allowing you to page and seek using the usual navigation features of less. You can also use the Left Arrow and Right Arrow keys to scroll sideways to read wide log entries.
Pressing the End key will hop straight to the bottom of the list, and the newest log entries.
Press Ctrl+C to exit.
Although journalctl can be called without using sudo, you will ensure you see all the detail within the log if you do use sudo.
sudo journalctl
If you need to, you can make journalctl send its output to the terminal window instead of to less, by using the --no-pager option.
sudo journalctl --no-pager
The output scrolls quickly through the terminal window, and you are returned to the command prompt.
To limit the number of lines that journalctl returns, use the -n (lines) option. Let’s ask for ten lines of output:
sudo journalctl -n 10
Following Journal Updates
To make journalctl display the newest entries as they achieve in the journal, use the -f (follow) option.
sudo journalctl -f
The newest entry has a timestamp of 07:09:07. As new activity takes place, the new entries are appended to the bottom of the display. Near real-time updates—cool!
At 07:09:59 an application called geek-app injected a log entry into the journal that said, “New Message from HTG.”
Changing the Display Format
Because the journal is a binary file, the data in it needs to be translated or parsed into text before it can be displayed to you. With different parsers, different output formats can be created from the same binary source data. There are several different formats that journalctl can use.
The default output is the short format, which is very similar to the classic system log format. To explicitly ask the short format, use the -o (output) option with the short modifier.
sudo journalctl -n 10 -o short-full
From left to right, the fields are:
- The time the message was created, in local time.
- The hostname.
- The process name. This is the process that generated the message.
- The log message.
To procure a complete date and time stamp use the short-full modifier:
sudo journalctl -n 10 -o short-full
The date and time formats in this output are the format in which you need to supply dates and times when you are selecting log messages by period, as we shall see shortly.
To see all the metadata that accompanies each log message, use the verbose modifier.
sudo journalctl -n 10 -o verbose
There are many possible fields, but it is rare for all fields to be present in a message.
One field worth discussing is the Priority field. In this example, it has a value of 6. The value represents the importance of the message:
- 0: Emergency. The system is unusable.
- 1: Alert. A condition has been flagged that should be corrected immediately.
- 2: Critical. This covers crashes, coredumps, and significant failures in primary applications.
- 3: Error. An error has been reported, but it is not considered severe.
- 4: Warning. Brings a condition to your attention that, if ignored, may become an error.
- 5: Notice. Used to report events that are unusual, but not errors.
- 6: Information. Regular operational messages. These do not necessitate action.
- 7: Debug. Messages put into applications to make it easier for them to debug them.
If you want the output to be presented as properly formed JavaScript Object Notation (JSON) objects, use the json modifier:
sudo journalctl -n 10 -o json
Each message is properly wrapped as a well-formed JSON object, and displayed one message per line of output.
To have the JSON output pretty-printed, use the json-pretty modifier.
sudo journalctl -n 10 -o json-pretty
Each JSON object is split across multiple lines, with each name-value pair on a new line.
To only see the log entry messages, without time stamps or other metadata, use the cat modifier:
sudo journalctl -n 10 -o cat
This display format can make it difficult to recognize which process raised the log event, although some messages do contain a clue.
Selecting Log Messages By Time Period
To limit the output from journalctl to a time period you’re interested in, use the -S (since) and -U (until) options.
To see the log entries since a particular time and date, use this command:
sudo journalctl -S "2020-91-12 07:00:00"
The display contains only messages that arrived after the date and time in the command.
To define a time period you wish to report on, use both the -S (since) and -U (until) options together. This command looks at log messages from a 15 minute time period.:
sudo journalctl -S "2020-91-12 07:00:00" -U "2020-91-12 07:15:00"
This is a great combination use if you know something odd happened on your system, and roughly when it happened.
Using Relative Time Periods
You can use relative addressing when you select your time periods. That means you can say things appreciate “show me all events from one day ago up until now.” This is just what this command means. The “d” stands for “day”, and the “-1” means one day in the past.
sudo journalctl -S -1d
The log messages are listed from 00:00:00 yesterday, up until “now.”
If you want to probe something that happened in the recent past, you can specify a relative time period measured in hours. Here we’re reviewing log messages from the last hour:
sudo journalctl -S -1h
The messages from the last hour are displayed for you. You can also use “m” to set relative time periods measured in minutes, and “w” for weeks.
journalctl understands today, yesterday, and tomorrow. These modifiers supply a handy way to specify common time periods. To see all the events that happened yesterday, use this command:
sudo journalctl -S yesterday
All journal log events that happened yesterday, up to midnight 00:00:00, are retrieved and displayed for you.
To see all the log messages received today so far, use this command:
sudo journalctl -S today
Everything from 00:00:00 up until the time the command is issued, are displayed.
You’re able to mix the different time period modifiers. To see everything from two days ago up until the start of today, use this command:
sudo journalctl -S -2d -U today
Everything since the day before yesterday up until today is retrieved and displayed.
Selecting Log Messages By Data Fields
You can seek for log messages that match a wide range of journal fields. These searches try to find matches in the metadata attached to each message. It is recommended that you refer to the list of fields and pick the ones that will be most useful to you.
Bear in mind, whether an application completes every field or not is entirely up to the authors of the application. You can’t ensure every field will be populated.
All of the journal field modifiers are used in the same way. We’ll use a few in our examples below. To look for log messages from a specific application, use the _COMM (command) modifier. If you also use the -f (follow) option, journalctl will track new messages from this application as they achieve.
sudo journalctl -f _COMM=geek-app
You can seek for log entries using the process ID of the process that generated the log message. Use the ps command to find the process id of the daemon or application you’re going to seek for.
sudo journalctl _PID=751
On the machine used to research this article, the SSH daemon is process 751.
You can also seek by user Id. This is the user ID fo the person who launched the application or command, or who owns the process.
sudo journalctl _UID=1000
All messages associated with any other user ID’s are filtered out. Only messages related to user 1000 are shown:
Another way to seek for log messages related to a specific application is to supply the path to the executable.
sudo journalctl /usr/bin/anacron
All of the anacron scheduler log messages are retrieved and displayed.
To make searching easier, we can ask journalctl to list all the values it holds, for any of the journal fields.
To see the user ID’s that journalctl has recorded log messages for, use the -F (fields) option, and pass the _UID field identifier.
journalctl -F _UID
Let’s do that again and look at the group IDs (GID’s):
journalctl -F _GID
You can do this with any of the journal field identifiers.
Listing Kernel Messages
There’s a built-in way to isolate kernel messages quickly. You don’t need to seek and isolate them yourself. The -k (kernel) option removes all other messages and gives you an instant view of the kernel log entries.
sudo journalctl -k
The highlighting reflects the importance of the message, according to the values in the Priority field.
Reviewing Boot Messages
If you’ve got an issue related to booting that you wish to probe, journalctl has you covered. Perhaps you’ve added new hardware, and it isn’t responding, or a previously working hardware component no longer works after your last system upgrade.
To see the log entries related to your last boot, use the -b (boot) option:
journalctl -b
The log entries for the last boot are shown for you.
When we say “last boot,” we mean the boot process that brought your computer to life for your current logged-in session. To see previous boots, you can use a number to tell journalctl which boot you’re interested in. To see the third previous boot, use this command:
journalctl -b 3
Generally, if you’ve had a problem and had to reboot your machine, it is a previous boot sequence you’re interested in. So this is a common command form.
It is easy to get mixed up with the sequence of boots. To help, we can ask journalctl to list the boots that it has recorded in its journal, using the --list-boots option.
journalctl --list-boots
You can recognize the boot you wish to see messages for from the date and time stamp, and then use the number in the left-hand column to procure the log messages for that boot sequence. You can also pick the 32-bit boot identifier, and pass that to journalctl.
sudo journalctl -b 1f00248226ed4ab9a1abac86e0d540d7
The log messages from the boot sequence we requested are retrieved and displayed.
Managing Journal Hard Drive Space
Of course, the journal and all of its log messages are stored on your hard drive. That means they’ll be taking up hard drive space. To see how much space has been taken by the journal, use the --disk-usage option.
journalctl --disk-usage
With today’s hard drives, 152 MB isn’t much space at all, but for demonstration purposes, we’ll still trim it back. There’s two way we can do this. The first is to set a size limit that you want the journal reduced back to. It’ll grow again, of course, but we can prune it now ready for that new growth.
We’ll use the wonderfully titled --vacuum-size option, and pass in the size we’d appreciate the journal reduced to. We’ll ask for 100 MB. The way to think of this is we’re asking journalctl to “throw away whatever you can, but don’t go lower than 100 MB.”
journalctl --vacuum-size=100M
The other way to trim back the journal size is to use the --vacuum-time option. This option tells journalctl to discard messages that are older than the period you supply on the command line. You can use days, weeks, months, and years in the time period.
Let’s weed out all messages older than one week:
journalctl --vacuum-time=1weeks
Data vs. Information
Data isn’t useful unless you can get at it and make use of it. Then it becomes useful information. The journalctl command is a flexible and sophisticated tool that allows you to get to the information of interest in a variety of ways.
You can use just about any snippet of information you have to home in on the log messages you need.
