From 034dd340b08be1f2f0477ad16131d609f9dbd53c Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Sun, 30 Aug 2026 09:22:00 -0700 Subject: Merge tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing fixes from Steven Rostedt: - Fix error output of boot instance creation failure Currently if a boot instance creation fails, instead of printing out the name of the instance that failed, it prints "(null)". That is because it prints "cur_str" that had already been processed by strsep(). Print the saved name instead. While at it, print the error code of the failure. - Fix use-after-free for same named historgrams Histograms can be named so that they can be used in multiple events. But if the named histogram has a variable attached, the second event that uses the named histogram which duplicates it and needs to free the original after duplication leaves the old variable in place and still visible. If another histogram uses than variable, it will use the stale one which will try to reference the freed duplicate histogram and crash the kernel. Free the duplicate variables along with the duplicated histogram data. - Check return value of kthread_run() in event self test The events self tests uses a kthread for testing but does not check if it succeeded in creating a kthread. If the kthread creation were to fail, the code will still try to call kthread_stop() on the error returned. - Fix race between reading trace_pipe and updating subbuffer size If a user is reading the trace_pipe file at the same time they update the ring buffer sub-buffer size, can cause the trace_pipe read to read stale data. Add trace_access_lock() around updating the ring buffer sub-buffer size. - Fix eventfs_inode on failure path in creation of the events directory In the creation of the "events" directory, if after allocating the eventfs_inode a failure is detected, it calls cleanup_ei() which calls free_ei(). The free_ei() will test if eventfs_inode being freed has no children. It is a bug if it does. But on the failure case of the creation of the "events" directory, the children lists have not yet been initialized and the free will trigger a warning because list_empty() on an uninitialized list returns false. Move the initialization into init_ei() where it makes more sense and makes sure that a created eventfs_inode has its lists initialized upon creation. - Check return value of kthread_run() in ftrace direct sample code The sample code that shows how to use the ftrace direct calls does not test the return of kthread_run() to see if it succeeds. Return a failure if the kthread_run() doesn't succeed. - Clear user events state on fork in case of alloc failure On fork, the child gets a pointer to the parent's user events state. It makes a copy of it then updates the child's pointer to it. But if the allocation fails, the duplication function leaves the child with a pointer to its parent's descriptor. When the child cleans up its data, it will free the parent's descriptor while the parent is still using it. In the duplication function, set the child's user_event_mm to NULL before testing if the allocation succeeded, and when it exits it will not free the parent's descriptor. - Fix retry exhaustion in simple ring buffer reader swap simple_ring_buffer_swap_reader_page() starts with retry set to 8 and post-decrements it only after a failed link replacement. On the final attempt, a successful replacement leaves retry at zero, while a failed replacement leaves it at -1. But the check for success expects the retry value to be non-zero and exits with an error on zero. This is the opposite result. Fix it. - Fail nicely when the remote swap_reader_page() returns an error Currently, if the swap_reader_page() of a remote buffer fails, it triggers a WARN_ON_ONCE() and continues normally. Instead, have it exit with an error and a pr_warn() print instead of a full WARNING. * tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: ring-buffer: Stop remote reader update when page swap fails tracing: Fix retry exhaustion in simple ring buffer reader swap tracing/user_events: Clear copied tracing state before fork duplication samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-multi-modify samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-modify eventfs: Initialize ei->children and ei->list in init_ei() tracing: Fix use-after-free in trace_pipe read on sub-buffer order change tracing: Fix crash passing ERR_PTR to kthread_stop() tracing: Fix use-after-free with same-name named triggers tracing: Fix logged instance name on creation failure --- Documentation/networking/netconsole.rst | 462 ++++++++++++++++++++++++++++++++ 1 file changed, 462 insertions(+) create mode 100644 Documentation/networking/netconsole.rst (limited to 'Documentation/networking/netconsole.rst') diff --git a/Documentation/networking/netconsole.rst b/Documentation/networking/netconsole.rst new file mode 100644 index 000000000..4ab5d7b05 --- /dev/null +++ b/Documentation/networking/netconsole.rst @@ -0,0 +1,462 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========== +Netconsole +========== + + +started by Ingo Molnar , 2001.09.17 + +2.6 port and netpoll api by Matt Mackall , Sep 9 2003 + +IPv6 support by Cong Wang , Jan 1 2013 + +Extended console support by Tejun Heo , May 1 2015 + +Release prepend support by Breno Leitao , Jul 7 2023 + +Userdata append support by Matthew Wood , Jan 22 2024 + +Sysdata append support by Breno Leitao , Jan 15 2025 + +Introduction: +============= + +This module logs kernel printk messages over UDP allowing debugging of +problem where disk logging fails and serial consoles are impractical. + +It can be used either built-in or as a module. As a built-in, +netconsole initializes immediately after NIC cards and will bring up +the specified interface as soon as possible. While this doesn't allow +capture of early kernel panics, it does capture most of the boot +process. + +Sender and receiver configuration: +================================== + +It takes a string configuration parameter "netconsole" in the +following format:: + + netconsole=[+][r][src-port]@[src-ip]/[],[tgt-port]@/[tgt-macaddr] + + where + + if present, enable extended console support + r if present, prepend kernel version (release) to the message + src-port source for UDP packets (defaults to 6665) + src-ip source IP to use (interface address) + dev network interface name (eth0) or MAC address + tgt-port port for logging agent (6666) + tgt-ip IP address for logging agent + tgt-macaddr ethernet MAC address for logging agent (broadcast) + +Examples:: + + linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc + +or:: + + insmod netconsole netconsole=@/,@10.0.0.2/ + +or using IPv6:: + + insmod netconsole netconsole=@/,@fd00:1:2:3::1/ + +or using a MAC address to select the egress interface:: + + linux netconsole=4444@10.0.0.1/22:33:44:55:66:77,9353@10.0.0.2/12:34:56:78:9a:bc + +It also supports logging to multiple remote agents by specifying +parameters for the multiple agents separated by semicolons and the +complete string enclosed in "quotes", thusly:: + + modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" + +Built-in netconsole starts immediately after the TCP stack is +initialized and attempts to bring up the supplied dev at the supplied +address. + +The remote host has several options to receive the kernel messages, +for example: + +1) syslogd + +2) netcat + + On distributions using a BSD-based netcat version (e.g. Fedora, + openSUSE and Ubuntu) the listening port must be specified without + the -p switch:: + + nc -u -l -p ' / 'nc -u -l + + or:: + + netcat -u -l -p ' / 'netcat -u -l + +3) socat + +:: + + socat udp-recv: - + +Dynamic reconfiguration: +======================== + +Dynamic reconfigurability is a useful addition to netconsole that enables +remote logging targets to be dynamically added, removed, or have their +parameters reconfigured at runtime from a configfs-based userspace interface. + +To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the +netconsole module (or kernel, if netconsole is built-in). + +Some examples follow (where configfs is mounted at the /sys/kernel/config +mountpoint). + +To add a remote logging target (target names can be arbitrary):: + + cd /sys/kernel/config/netconsole/ + mkdir target1 + +Note that newly created targets have default parameter values (as mentioned +above) and are disabled by default -- they must first be enabled by writing +"1" to the "enabled" attribute (usually after setting parameters accordingly) +as described below. + +To remove a target:: + + rmdir /sys/kernel/config/netconsole/othertarget/ + +The interface exposes these parameters of a netconsole target to userspace: + + =============== ================================= ============ + enabled Is this target currently enabled? (read-write) + extended Extended mode enabled (read-write) + release Prepend kernel release to message (read-write) + dev_name Local network interface name (read-write) + local_port Source UDP port to use (read-write) + remote_port Remote agent's UDP port (read-write) + local_ip Source IP address to use (read-write) + remote_ip Remote agent's IP address (read-write) + local_mac Local interface's MAC address (read-only) + remote_mac Remote agent's MAC address (read-write) + transmit_errors Number of packet send errors (read-only) + =============== ================================= ============ + +The "enabled" attribute is also used to control whether the parameters of +a target can be updated or not -- you can modify the parameters of only +disabled targets (i.e. if "enabled" is 0). + +To update a target's parameters:: + + cat enabled # check if enabled is 1 + echo 0 > enabled # disable the target (if required) + echo eth2 > dev_name # set local interface + echo 10.0.0.4 > remote_ip # update some parameter + echo cb:a9:87:65:43:21 > remote_mac # update more parameters + echo 1 > enabled # enable target again + +You can also update the local interface dynamically. This is especially +useful if you want to use interfaces that have newly come up (and may not +have existed when netconsole was loaded / initialized). + +Netconsole targets defined at boot time (or module load time) with the +`netconsole=` param are assigned the name `cmdline`. For example, the +first target in the parameter is named `cmdline0`. You can control and modify +these targets by creating configfs directories with the matching name. + +Let's suppose you have two netconsole targets defined at boot time:: + + netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc + +You can modify these targets in runtime by creating the following targets:: + + mkdir cmdline0 + cat cmdline0/remote_ip + 10.0.0.2 + + mkdir cmdline1 + cat cmdline1/remote_ip + 10.0.0.3 + +Append User Data +---------------- + +Custom user data can be appended to the end of messages with netconsole +dynamic configuration enabled. User data entries can be modified without +changing the "enabled" attribute of a target. + +Directories (keys) under `userdata` are limited to 53 character length, and +data in `userdata//value` are limited to 200 bytes:: + + cd /sys/kernel/config/netconsole && mkdir cmdline0 + cd cmdline0 + mkdir userdata/foo + echo bar > userdata/foo/value + mkdir userdata/qux + echo baz > userdata/qux/value + +Messages will now include this additional user data:: + + echo "This is a message" > /dev/kmsg + +Sends:: + + 12,607,22085407756,-;This is a message + foo=bar + qux=baz + +Preview the userdata that will be appended with:: + + cd /sys/kernel/config/netconsole/cmdline0/userdata + for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done + +If a `userdata` entry is created but no data is written to the `value` file, +the entry will be omitted from netconsole messages:: + + cd /sys/kernel/config/netconsole && mkdir cmdline0 + cd cmdline0 + mkdir userdata/foo + echo bar > userdata/foo/value + mkdir userdata/qux + +The `qux` key is omitted since it has no value:: + + echo "This is a message" > /dev/kmsg + 12,607,22085407756,-;This is a message + foo=bar + +Delete `userdata` entries with `rmdir`:: + + rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux + +.. warning:: + When writing strings to user data values, input is broken up per line in + configfs store calls and this can cause confusing behavior:: + + mkdir userdata/testing + printf "val1\nval2" > userdata/testing/value + # userdata store value is called twice, first with "val1\n" then "val2" + # so "val2" is stored, being the last value stored + cat userdata/testing/value + val2 + + It is recommended to not write user data values with newlines. + +Task name auto population in userdata +------------------------------------- + +Inside the netconsole configfs hierarchy, there is a file called +`taskname_enabled` under the `userdata` directory. This file is used to enable +or disable the automatic task name population feature. This feature +automatically populates the current task name that is scheduled in the CPU +sneding the message. + +To enable task name auto-population:: + + echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled + +When this option is enabled, the netconsole messages will include an additional +line in the userdata field with the format `taskname=`. This allows +the receiver of the netconsole messages to easily find which application was +currently scheduled when that message was generated, providing extra context +for kernel messages and helping to categorize them. + +Example:: + + echo "This is a message" > /dev/kmsg + 12,607,22085407756,-;This is a message + taskname=echo + +In this example, the message was generated while "echo" was the current +scheduled process. + +Kernel release auto population in userdata +------------------------------------------ + +Within the netconsole configfs hierarchy, there is a file named `release_enabled` +located in the `userdata` directory. This file controls the kernel release +(version) auto-population feature, which appends the kernel release information +to userdata dictionary in every message sent. + +To enable the release auto-population:: + + echo 1 > /sys/kernel/config/netconsole/target1/userdata/release_enabled + +Example:: + + echo "This is a message" > /dev/kmsg + 12,607,22085407756,-;This is a message + release=6.14.0-rc6-01219-g3c027fbd941d + +.. note:: + + This feature provides the same data as the "release prepend" feature. + However, in this case, the release information is appended to the userdata + dictionary rather than being included in the message header. + + +CPU number auto population in userdata +-------------------------------------- + +Inside the netconsole configfs hierarchy, there is a file called +`cpu_nr` under the `userdata` directory. This file is used to enable or disable +the automatic CPU number population feature. This feature automatically +populates the CPU number that is sending the message. + +To enable the CPU number auto-population:: + + echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr + +When this option is enabled, the netconsole messages will include an additional +line in the userdata field with the format `cpu=`. This allows the +receiver of the netconsole messages to easily differentiate and demultiplex +messages originating from different CPUs, which is particularly useful when +dealing with parallel log output. + +Example:: + + echo "This is a message" > /dev/kmsg + 12,607,22085407756,-;This is a message + cpu=42 + +In this example, the message was sent by CPU 42. + +.. note:: + + If the user has set a conflicting `cpu` key in the userdata dictionary, + both keys will be reported, with the kernel-populated entry appearing after + the user one. For example:: + + # User-defined CPU entry + mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu + echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value + + Output might look like:: + + 12,607,22085407756,-;This is a message + cpu=1 + cpu=42 # kernel-populated value + + +Message ID auto population in userdata +-------------------------------------- + +Within the netconsole configfs hierarchy, there is a file named `msgid_enabled` +located in the `userdata` directory. This file controls the message ID +auto-population feature, which assigns a numeric id to each message sent to a +given target and appends the ID to userdata dictionary in every message sent. + +The message ID is generated using a per-target 32 bit counter that is +incremented for every message sent to the target. Note that this counter will +eventually wrap around after reaching uint32_t max value, so the message ID is +not globally unique over time. However, it can still be used by the target to +detect if messages were dropped before reaching the target by identifying gaps +in the sequence of IDs. + +It is important to distinguish message IDs from the message field. +Some kernel messages may never reach netconsole (for example, due to printk +rate limiting). Thus, a gap in cannot be solely relied upon to +indicate that a message was dropped during transmission, as it may never have +been sent via netconsole. The message ID, on the other hand, is only assigned +to messages that are actually transmitted via netconsole. + +Example:: + + echo "This is message #1" > /dev/kmsg + echo "This is message #2" > /dev/kmsg + 13,434,54928466,-;This is message #1 + msgid=1 + 13,435,54934019,-;This is message #2 + msgid=2 + + +Extended console: +================= + +If '+' is prefixed to the configuration line or "extended" config file +is set to 1, extended console support is enabled. An example boot +param follows:: + + linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc + +Log messages are transmitted with extended metadata header in the +following format which is the same as /dev/kmsg:: + + ,,,; + +If 'r' (release) feature is enabled, the kernel release version is +prepended to the start of the message. Example:: + + 6.4.0,6,444,501151268,-;netconsole: network logging started + +Non printable characters in are escaped using "\xff" +notation. If the message contains optional dictionary, verbatim +newline is used as the delimiter. + +If a message doesn't fit in certain number of bytes (currently 1000), +the message is split into multiple fragments by netconsole. These +fragments are transmitted with "ncfrag" header field added:: + + ncfrag=/ + +For example, assuming a lot smaller chunk size, a message "the first +chunk, the 2nd chunk." may be split as follows:: + + 6,416,1758426,-,ncfrag=0/31;the first chunk, + 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. + +Miscellaneous notes: +==================== + +.. Warning:: + + the default target ethernet setting uses the broadcast + ethernet address to send packets, which can cause increased load on + other systems on the same ethernet segment. + +.. Tip:: + + some LAN switches may be configured to suppress ethernet broadcasts + so it is advised to explicitly specify the remote agents' MAC addresses + from the config parameters passed to netconsole. + +.. Tip:: + + to find out the MAC address of, say, 10.0.0.2, you may try using:: + + ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 + +.. Tip:: + + in case the remote logging agent is on a separate LAN subnet than + the sender, it is suggested to try specifying the MAC address of the + default gateway (you may use /sbin/route -n to find it out) as the + remote MAC address instead. + +.. note:: + + the network device (eth1 in the above case) can run any kind + of other network traffic, netconsole is not intrusive. Netconsole + might cause slight delays in other traffic if the volume of kernel + messages is high, but should have no other impact. + +.. note:: + + if you find that the remote logging agent is not receiving or + printing all messages from the sender, it is likely that you have set + the "console_loglevel" parameter (on the sender) to only send high + priority messages to the console. You can change this at runtime using:: + + dmesg -n 8 + + or by specifying "debug" on the kernel command line at boot, to send + all kernel messages to the console. A specific value for this parameter + can also be set using the "loglevel" kernel boot option. See the + dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst + for details. + +Netconsole was designed to be as instantaneous as possible, to +enable the logging of even the most critical kernel bugs. It works +from IRQ contexts as well, and does not enable interrupts while +sending packets. Due to these unique needs, configuration cannot +be more automatic, and some fundamental limitations will remain: +only IP networks, UDP packets and ethernet devices are supported. -- cgit v1.3.1