x86/kprobes: Fix crash when probing CS CALL instructionsgrafted

When using eBPF to probe CS CALL instructions within a function,
a crash can be triggered.

The eBPF tool probes offset 257 of the __hrtimer_run_queues()
function:

<__hrtimer_run_queues+249>:  nopl   0x0(%rax,%rax,1)
<__hrtimer_run_queues+254>:  mov    %r14,%rdi
<__hrtimer_run_queues+257>:  cs call <__x86_indirect_thunk_r12>
<__hrtimer_run_queues+263>:  mov    %eax,%r12d
<__hrtimer_run_queues+266>:  xchg   %ax,%ax
<__hrtimer_run_queues+268>:  mov    %r13,%rdi

Which triggers this crash:

  BUG: unable to handle page fault for address: 00000000000f41c9
  #PF: supervisor write access in kernel mode
  #PF: error_code(0x0002) - not-present page
  PGD 0 P4D 0
  Oops: 0002 [#1] SMP NOPTI
  CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Tainted: P
  RIP: 0010:__hrtimer_run_queues+0x106/0x230

Note that __hrtimer_run_queues+0x106 is __hrtimer_run_queues+262, which is
at the 6th byte of the above CS CALL instruction. Since the CS CALL
instruction occupies 6 bytes, the exception occurred in the middle of that
call instruction.

The root cause is that when using eBPF tools to probe in the middle of a
function, a kprobe with INT3 is used as the underlying implementation.

During single-step emulation of the original CALL instruction,
int3_emulate_call() assumes that the probed CALL instruction is 5 bytes
long. However, the actual CS-prefixed CALL instruction occupies 6 bytes,
so it constructs an incorrect exception return address. When the CPU
returns from the kprobe handler, the next instruction to be executed is at
the address of the last byte of that CS CALL instruction. Coincidentally,
starting from that address, the CPU fetches and decodes a completely
different instruction, which ultimately triggers a kernel crash.

Fix the issue by using the actual instruction length obtained from
the instruction decoder when constructing the exception return
address, rather than relying on the hardcoded CALL_INSN_SIZE macro.

[ mingo: Refined the changelog ]

Fixes: 6256e668b7af ("x86/kprobes: Use int3 instead of debug trap for single-step")
Suggested-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Jinke Han <jinkehan@didiglobal.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Acked-by: Yafang Shao <laoar.shao@gmail.com>
Acked-by: Borislav Petkov <bp@alien8.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://patch.msgid.link/20260908073742.GA10517@didi-ThinkCentre-M920t-N000

1 files changed, 263 insertions, 0 deletions

diff --git a/fs/ecryptfs/read_write.c b/fs/ecryptfs/read_write.c
new file mode 100644
index 000000000..b3b451c2b
--- /dev/null
+++ b/fs/ecryptfs/read_write.c
@@ -0,0 +1,263 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * eCryptfs: Linux filesystem encryption layer
+ *
+ * Copyright (C) 2007 International Business Machines Corp.
+ *   Author(s): Michael A. Halcrow <mahalcro@us.ibm.com>
+ */
+
+#include <linux/fs.h>
+#include <linux/pagemap.h>
+#include <linux/sched/signal.h>
+
+#include "ecryptfs_kernel.h"
+
+/**
+ * ecryptfs_write_lower
+ * @ecryptfs_inode: The eCryptfs inode
+ * @data: Data to write
+ * @offset: Byte offset in the lower file to which to write the data
+ * @size: Number of bytes from @data to write at @offset in the lower
+ *        file
+ *
+ * Write data to the lower file.
+ *
+ * Returns bytes written on success; less than zero on error
+ */
+int ecryptfs_write_lower(struct inode *ecryptfs_inode, char *data,
+			 loff_t offset, size_t size)
+{
+	struct file *lower_file;
+	ssize_t rc;
+
+	lower_file = ecryptfs_inode_to_private(ecryptfs_inode)->lower_file;
+	if (!lower_file)
+		return -EIO;
+	rc = kernel_write(lower_file, data, size, &offset);
+	mark_inode_dirty_sync(ecryptfs_inode);
+	return rc;
+}
+
+/**
+ * ecryptfs_write_lower_page_segment
+ * @ecryptfs_inode: The eCryptfs inode
+ * @folio_for_lower: The folio containing the data to be written to the
+ *                  lower file
+ * @offset_in_page: The offset in the @folio_for_lower from which to
+ *                  start writing the data
+ * @size: The amount of data from @folio_for_lower to write to the
+ *        lower file
+ *
+ * Determines the byte offset in the file for the given page and
+ * offset within the page, maps the page, and makes the call to write
+ * the contents of @folio_for_lower to the lower inode.
+ *
+ * Returns zero on success; non-zero otherwise
+ */
+int ecryptfs_write_lower_page_segment(struct inode *ecryptfs_inode,
+				      struct folio *folio_for_lower,
+				      size_t offset_in_page, size_t size)
+{
+	char *virt;
+	loff_t offset;
+	int rc;
+
+	offset = (loff_t)folio_for_lower->index * PAGE_SIZE + offset_in_page;
+	virt = kmap_local_folio(folio_for_lower, 0);
+	rc = ecryptfs_write_lower(ecryptfs_inode, virt, offset, size);
+	if (rc > 0)
+		rc = 0;
+	kunmap_local(virt);
+	return rc;
+}
+
+/**
+ * ecryptfs_write
+ * @ecryptfs_inode: The eCryptfs file into which to write
+ * @data: Virtual address where data to write is located
+ * @offset: Offset in the eCryptfs file at which to begin writing the
+ *          data from @data
+ * @size: The number of bytes to write from @data
+ *
+ * Write an arbitrary amount of data to an arbitrary location in the
+ * eCryptfs inode page cache. This is done on a page-by-page, and then
+ * by an extent-by-extent, basis; individual extents are encrypted and
+ * written to the lower page cache (via VFS writes). This function
+ * takes care of all the address translation to locations in the lower
+ * filesystem; it also handles truncate events, writing out zeros
+ * where necessary.
+ *
+ * Returns zero on success; non-zero otherwise
+ */
+int ecryptfs_write(struct inode *ecryptfs_inode, char *data, loff_t offset,
+		   size_t size)
+{
+	struct ecryptfs_crypt_stat *crypt_stat;
+	char *ecryptfs_page_virt;
+	loff_t ecryptfs_file_size = i_size_read(ecryptfs_inode);
+	loff_t data_offset = 0;
+	loff_t pos;
+	int rc = 0;
+
+	crypt_stat = &ecryptfs_inode_to_private(ecryptfs_inode)->crypt_stat;
+	/*
+	 * if we are writing beyond current size, then start pos
+	 * at the current size - we'll fill in zeros from there.
+	 */
+	if (offset > ecryptfs_file_size)
+		pos = ecryptfs_file_size;
+	else
+		pos = offset;
+	while (pos < (offset + size)) {
+		struct folio *ecryptfs_folio;
+		pgoff_t ecryptfs_page_idx = (pos >> PAGE_SHIFT);
+		size_t start_offset_in_page = (pos & ~PAGE_MASK);
+		size_t num_bytes = (PAGE_SIZE - start_offset_in_page);
+		loff_t total_remaining_bytes = ((offset + size) - pos);
+
+		if (fatal_signal_pending(current)) {
+			rc = -EINTR;
+			break;
+		}
+
+		if (num_bytes > total_remaining_bytes)
+			num_bytes = total_remaining_bytes;
+		if (pos < offset) {
+			/* remaining zeros to write, up to destination offset */
+			loff_t total_remaining_zeros = (offset - pos);
+
+			if (num_bytes > total_remaining_zeros)
+				num_bytes = total_remaining_zeros;
+		}
+		ecryptfs_folio = read_mapping_folio(ecryptfs_inode->i_mapping,
+				ecryptfs_page_idx, NULL);
+		if (IS_ERR(ecryptfs_folio)) {
+			rc = PTR_ERR(ecryptfs_folio);
+			printk(KERN_ERR "%s: Error getting page at "
+			       "index [%ld] from eCryptfs inode "
+			       "mapping; rc = [%d]\n", __func__,
+			       ecryptfs_page_idx, rc);
+			goto out;
+		}
+		folio_lock(ecryptfs_folio);
+		ecryptfs_page_virt = kmap_local_folio(ecryptfs_folio, 0);
+
+		/*
+		 * pos: where we're now writing, offset: where the request was
+		 * If current pos is before request, we are filling zeros
+		 * If we are at or beyond request, we are writing the *data*
+		 * If we're in a fresh page beyond eof, zero it in either case
+		 */
+		if (pos < offset || !start_offset_in_page) {
+			/* We are extending past the previous end of the file.
+			 * Fill in zero values to the end of the page */
+			memset(((char *)ecryptfs_page_virt
+				+ start_offset_in_page), 0,
+				PAGE_SIZE - start_offset_in_page);
+		}
+
+		/* pos >= offset, we are now writing the data request */
+		if (pos >= offset) {
+			memcpy(((char *)ecryptfs_page_virt
+				+ start_offset_in_page),
+			       (data + data_offset), num_bytes);
+			data_offset += num_bytes;
+		}
+		kunmap_local(ecryptfs_page_virt);
+		flush_dcache_folio(ecryptfs_folio);
+		folio_mark_uptodate(ecryptfs_folio);
+		folio_unlock(ecryptfs_folio);
+		if (crypt_stat->flags & ECRYPTFS_ENCRYPTED)
+			rc = ecryptfs_encrypt_page(ecryptfs_folio);
+		else
+			rc = ecryptfs_write_lower_page_segment(ecryptfs_inode,
+						ecryptfs_folio,
+						start_offset_in_page,
+						data_offset);
+		folio_put(ecryptfs_folio);
+		if (rc) {
+			printk(KERN_ERR "%s: Error encrypting "
+			       "page; rc = [%d]\n", __func__, rc);
+			goto out;
+		}
+		pos += num_bytes;
+	}
+	if (pos > ecryptfs_file_size) {
+		i_size_write(ecryptfs_inode, pos);
+		if (crypt_stat->flags & ECRYPTFS_ENCRYPTED) {
+			int rc2;
+
+			rc2 = ecryptfs_write_inode_size_to_metadata(
+								ecryptfs_inode);
+			if (rc2) {
+				printk(KERN_ERR	"Problem with "
+				       "ecryptfs_write_inode_size_to_metadata; "
+				       "rc = [%d]\n", rc2);
+				if (!rc)
+					rc = rc2;
+				goto out;
+			}
+		}
+	}
+out:
+	return rc;
+}
+
+/**
+ * ecryptfs_read_lower
+ * @data: The read data is stored here by this function
+ * @offset: Byte offset in the lower file from which to read the data
+ * @size: Number of bytes to read from @offset of the lower file and
+ *        store into @data
+ * @ecryptfs_inode: The eCryptfs inode
+ *
+ * Read @size bytes of data at byte offset @offset from the lower
+ * inode into memory location @data.
+ *
+ * Returns bytes read on success; 0 on EOF; less than zero on error
+ */
+int ecryptfs_read_lower(char *data, loff_t offset, size_t size,
+			struct inode *ecryptfs_inode)
+{
+	struct file *lower_file;
+	lower_file = ecryptfs_inode_to_private(ecryptfs_inode)->lower_file;
+	if (!lower_file)
+		return -EIO;
+	return kernel_read(lower_file, data, size, &offset);
+}
+
+/**
+ * ecryptfs_read_lower_page_segment
+ * @folio_for_ecryptfs: The folio into which data for eCryptfs will be
+ *                     written
+ * @page_index: Page index in @page_for_ecryptfs from which to start
+ *		writing
+ * @offset_in_page: Offset in @page_for_ecryptfs from which to start
+ *                  writing
+ * @size: The number of bytes to write into @page_for_ecryptfs
+ * @ecryptfs_inode: The eCryptfs inode
+ *
+ * Determines the byte offset in the file for the given page and
+ * offset within the page, maps the page, and makes the call to read
+ * the contents of @page_for_ecryptfs from the lower inode.
+ *
+ * Returns zero on success; non-zero otherwise
+ */
+int ecryptfs_read_lower_page_segment(struct folio *folio_for_ecryptfs,
+				     pgoff_t page_index,
+				     size_t offset_in_page, size_t size,
+				     struct inode *ecryptfs_inode)
+{
+	char *virt;
+	loff_t offset;
+	int rc;
+
+	offset = (loff_t)page_index * PAGE_SIZE + offset_in_page;
+	virt = kmap_local_folio(folio_for_ecryptfs, 0);
+	rc = ecryptfs_read_lower(virt, offset, size, ecryptfs_inode);
+	if (rc > 0)
+		rc = 0;
+	kunmap_local(virt);
+	flush_dcache_folio(folio_for_ecryptfs);
+	return rc;
+}