From 034dd340b08be1f2f0477ad16131d609f9dbd53c Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Sun, 30 Aug 2026 09:22:00 -0700 Subject: Merge tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing fixes from Steven Rostedt: - Fix error output of boot instance creation failure Currently if a boot instance creation fails, instead of printing out the name of the instance that failed, it prints "(null)". That is because it prints "cur_str" that had already been processed by strsep(). Print the saved name instead. While at it, print the error code of the failure. - Fix use-after-free for same named historgrams Histograms can be named so that they can be used in multiple events. But if the named histogram has a variable attached, the second event that uses the named histogram which duplicates it and needs to free the original after duplication leaves the old variable in place and still visible. If another histogram uses than variable, it will use the stale one which will try to reference the freed duplicate histogram and crash the kernel. Free the duplicate variables along with the duplicated histogram data. - Check return value of kthread_run() in event self test The events self tests uses a kthread for testing but does not check if it succeeded in creating a kthread. If the kthread creation were to fail, the code will still try to call kthread_stop() on the error returned. - Fix race between reading trace_pipe and updating subbuffer size If a user is reading the trace_pipe file at the same time they update the ring buffer sub-buffer size, can cause the trace_pipe read to read stale data. Add trace_access_lock() around updating the ring buffer sub-buffer size. - Fix eventfs_inode on failure path in creation of the events directory In the creation of the "events" directory, if after allocating the eventfs_inode a failure is detected, it calls cleanup_ei() which calls free_ei(). The free_ei() will test if eventfs_inode being freed has no children. It is a bug if it does. But on the failure case of the creation of the "events" directory, the children lists have not yet been initialized and the free will trigger a warning because list_empty() on an uninitialized list returns false. Move the initialization into init_ei() where it makes more sense and makes sure that a created eventfs_inode has its lists initialized upon creation. - Check return value of kthread_run() in ftrace direct sample code The sample code that shows how to use the ftrace direct calls does not test the return of kthread_run() to see if it succeeds. Return a failure if the kthread_run() doesn't succeed. - Clear user events state on fork in case of alloc failure On fork, the child gets a pointer to the parent's user events state. It makes a copy of it then updates the child's pointer to it. But if the allocation fails, the duplication function leaves the child with a pointer to its parent's descriptor. When the child cleans up its data, it will free the parent's descriptor while the parent is still using it. In the duplication function, set the child's user_event_mm to NULL before testing if the allocation succeeded, and when it exits it will not free the parent's descriptor. - Fix retry exhaustion in simple ring buffer reader swap simple_ring_buffer_swap_reader_page() starts with retry set to 8 and post-decrements it only after a failed link replacement. On the final attempt, a successful replacement leaves retry at zero, while a failed replacement leaves it at -1. But the check for success expects the retry value to be non-zero and exits with an error on zero. This is the opposite result. Fix it. - Fail nicely when the remote swap_reader_page() returns an error Currently, if the swap_reader_page() of a remote buffer fails, it triggers a WARN_ON_ONCE() and continues normally. Instead, have it exit with an error and a pr_warn() print instead of a full WARNING. * tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: ring-buffer: Stop remote reader update when page swap fails tracing: Fix retry exhaustion in simple ring buffer reader swap tracing/user_events: Clear copied tracing state before fork duplication samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-multi-modify samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-modify eventfs: Initialize ei->children and ei->list in init_ei() tracing: Fix use-after-free in trace_pipe read on sub-buffer order change tracing: Fix crash passing ERR_PTR to kthread_stop() tracing: Fix use-after-free with same-name named triggers tracing: Fix logged instance name on creation failure --- drivers/net/team/Kconfig | 77 + drivers/net/team/Makefile | 12 + drivers/net/team/team_core.c | 3269 +++++++++++++++++++++++++++++ drivers/net/team/team_mode_activebackup.c | 142 ++ drivers/net/team/team_mode_broadcast.c | 73 + drivers/net/team/team_mode_loadbalance.c | 698 ++++++ drivers/net/team/team_mode_random.c | 64 + drivers/net/team/team_mode_roundrobin.c | 75 + drivers/net/team/team_nl.c | 60 + drivers/net/team/team_nl.h | 30 + 10 files changed, 4500 insertions(+) create mode 100644 drivers/net/team/Kconfig create mode 100644 drivers/net/team/Makefile create mode 100644 drivers/net/team/team_core.c create mode 100644 drivers/net/team/team_mode_activebackup.c create mode 100644 drivers/net/team/team_mode_broadcast.c create mode 100644 drivers/net/team/team_mode_loadbalance.c create mode 100644 drivers/net/team/team_mode_random.c create mode 100644 drivers/net/team/team_mode_roundrobin.c create mode 100644 drivers/net/team/team_nl.c create mode 100644 drivers/net/team/team_nl.h (limited to 'drivers/net/team') diff --git a/drivers/net/team/Kconfig b/drivers/net/team/Kconfig new file mode 100644 index 000000000..4d449845d --- /dev/null +++ b/drivers/net/team/Kconfig @@ -0,0 +1,77 @@ +# SPDX-License-Identifier: GPL-2.0-only +menuconfig NET_TEAM + tristate "Ethernet team driver support" + help + This allows one to create virtual interfaces that teams together + multiple ethernet devices. + + Team devices can be added using the "ip" command from the + iproute2 package: + + "ip link add link [ address MAC ] [ NAME ] type team" + + To compile this driver as a module, choose M here: the module + will be called team. + +if NET_TEAM + +config NET_TEAM_MODE_BROADCAST + tristate "Broadcast mode support" + depends on NET_TEAM + help + Basic mode where packets are transmitted always by all suitable ports. + + All added ports are setup to have team's device address. + + To compile this team mode as a module, choose M here: the module + will be called team_mode_broadcast. + +config NET_TEAM_MODE_ROUNDROBIN + tristate "Round-robin mode support" + depends on NET_TEAM + help + Basic mode where port used for transmitting packets is selected in + round-robin fashion using packet counter. + + All added ports are setup to have team's device address. + + To compile this team mode as a module, choose M here: the module + will be called team_mode_roundrobin. + +config NET_TEAM_MODE_RANDOM + tristate "Random mode support" + depends on NET_TEAM + help + Basic mode where port used for transmitting packets is selected + randomly. + + All added ports are setup to have team's device address. + + To compile this team mode as a module, choose M here: the module + will be called team_mode_random. + +config NET_TEAM_MODE_ACTIVEBACKUP + tristate "Active-backup mode support" + depends on NET_TEAM + help + Only one port is active at a time and the rest of ports are used + for backup. + + Mac addresses of ports are not modified. Userspace is responsible + to do so. + + To compile this team mode as a module, choose M here: the module + will be called team_mode_activebackup. + +config NET_TEAM_MODE_LOADBALANCE + tristate "Load-balance mode support" + depends on NET_TEAM + help + This mode provides load balancing functionality. Tx port selection + is done using BPF function set up from userspace (bpf_hash_func + option) + + To compile this team mode as a module, choose M here: the module + will be called team_mode_loadbalance. + +endif # NET_TEAM diff --git a/drivers/net/team/Makefile b/drivers/net/team/Makefile new file mode 100644 index 000000000..7a5aa20d2 --- /dev/null +++ b/drivers/net/team/Makefile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for the network team driver +# + +team-y:= team_core.o team_nl.o +obj-$(CONFIG_NET_TEAM) += team.o +obj-$(CONFIG_NET_TEAM_MODE_BROADCAST) += team_mode_broadcast.o +obj-$(CONFIG_NET_TEAM_MODE_ROUNDROBIN) += team_mode_roundrobin.o +obj-$(CONFIG_NET_TEAM_MODE_RANDOM) += team_mode_random.o +obj-$(CONFIG_NET_TEAM_MODE_ACTIVEBACKUP) += team_mode_activebackup.o +obj-$(CONFIG_NET_TEAM_MODE_LOADBALANCE) += team_mode_loadbalance.o diff --git a/drivers/net/team/team_core.c b/drivers/net/team/team_core.c new file mode 100644 index 000000000..beffbe450 --- /dev/null +++ b/drivers/net/team/team_core.c @@ -0,0 +1,3269 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * drivers/net/team/team.c - Network team device driver + * Copyright (c) 2011 Jiri Pirko + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "team_nl.h" + +#define DRV_NAME "team" + + +/********** + * Helpers + **********/ + +static struct team_port *team_port_get_rtnl(const struct net_device *dev) +{ + struct team_port *port = rtnl_dereference(dev->rx_handler_data); + + return netif_is_team_port(dev) ? port : NULL; +} + +/* + * Since the ability to change device address for open port device is tested in + * team_port_add, this function can be called without control of return value + */ +static int __set_port_dev_addr(struct net_device *port_dev, + const unsigned char *dev_addr) +{ + struct sockaddr_storage addr; + + memcpy(addr.__data, dev_addr, port_dev->addr_len); + addr.ss_family = port_dev->type; + return dev_set_mac_address(port_dev, &addr, NULL); +} + +static int team_port_set_orig_dev_addr(struct team_port *port) +{ + return __set_port_dev_addr(port->dev, port->orig.dev_addr); +} + +static int team_port_set_team_dev_addr(struct team *team, + struct team_port *port) +{ + return __set_port_dev_addr(port->dev, netdev_from_priv(team)->dev_addr); +} + +int team_modeop_port_enter(struct team *team, struct team_port *port) +{ + return team_port_set_team_dev_addr(team, port); +} +EXPORT_SYMBOL(team_modeop_port_enter); + +void team_modeop_port_change_dev_addr(struct team *team, + struct team_port *port) +{ + team_port_set_team_dev_addr(team, port); +} +EXPORT_SYMBOL(team_modeop_port_change_dev_addr); + +static void team_lower_state_changed(struct team_port *port) +{ + struct netdev_lag_lower_state_info info; + + info.link_up = port->linkup; + info.tx_enabled = team_port_tx_enabled(port); + netdev_lower_state_changed(port->dev, &info); +} + +static void team_refresh_port_linkup(struct team_port *port) +{ + bool new_linkup = port->user.linkup_enabled ? port->user.linkup : + port->state.linkup; + + if (port->linkup != new_linkup) { + port->linkup = new_linkup; + team_lower_state_changed(port); + } +} + + +/******************* + * Options handling + *******************/ + +struct team_option_inst { /* One for each option instance */ + struct list_head list; + struct list_head tmp_list; + struct team_option *option; + struct team_option_inst_info info; + bool changed; + bool removed; +}; + +static struct team_option *__team_find_option(struct team *team, + const char *opt_name) +{ + struct team_option *option; + + list_for_each_entry(option, &team->option_list, list) { + if (strcmp(option->name, opt_name) == 0) + return option; + } + return NULL; +} + +static void __team_option_inst_del(struct team_option_inst *opt_inst) +{ + list_del(&opt_inst->list); + kfree(opt_inst); +} + +static void __team_option_inst_del_option(struct team *team, + struct team_option *option) +{ + struct team_option_inst *opt_inst, *tmp; + + list_for_each_entry_safe(opt_inst, tmp, &team->option_inst_list, list) { + if (opt_inst->option == option) + __team_option_inst_del(opt_inst); + } +} + +static int __team_option_inst_add(struct team *team, struct team_option *option, + struct team_port *port) +{ + struct team_option_inst *opt_inst; + unsigned int array_size; + unsigned int i; + + array_size = option->array_size; + if (!array_size) + array_size = 1; /* No array but still need one instance */ + + for (i = 0; i < array_size; i++) { + opt_inst = kmalloc_obj(*opt_inst); + if (!opt_inst) + return -ENOMEM; + opt_inst->option = option; + opt_inst->info.port = port; + opt_inst->info.array_index = i; + opt_inst->changed = true; + opt_inst->removed = false; + list_add_tail(&opt_inst->list, &team->option_inst_list); + if (option->init) + option->init(team, &opt_inst->info); + + } + return 0; +} + +static int __team_option_inst_add_option(struct team *team, + struct team_option *option) +{ + int err; + + if (!option->per_port) { + err = __team_option_inst_add(team, option, NULL); + if (err) + goto inst_del_option; + } + return 0; + +inst_del_option: + __team_option_inst_del_option(team, option); + return err; +} + +static void __team_option_inst_mark_removed_option(struct team *team, + struct team_option *option) +{ + struct team_option_inst *opt_inst; + + list_for_each_entry(opt_inst, &team->option_inst_list, list) { + if (opt_inst->option == option) { + opt_inst->changed = true; + opt_inst->removed = true; + } + } +} + +static void __team_option_inst_del_port(struct team *team, + struct team_port *port) +{ + struct team_option_inst *opt_inst, *tmp; + + list_for_each_entry_safe(opt_inst, tmp, &team->option_inst_list, list) { + if (opt_inst->option->per_port && + opt_inst->info.port == port) + __team_option_inst_del(opt_inst); + } +} + +static int __team_option_inst_add_port(struct team *team, + struct team_port *port) +{ + struct team_option *option; + int err; + + list_for_each_entry(option, &team->option_list, list) { + if (!option->per_port) + continue; + err = __team_option_inst_add(team, option, port); + if (err) + goto inst_del_port; + } + return 0; + +inst_del_port: + __team_option_inst_del_port(team, port); + return err; +} + +static void __team_option_inst_mark_removed_port(struct team *team, + struct team_port *port) +{ + struct team_option_inst *opt_inst; + + list_for_each_entry(opt_inst, &team->option_inst_list, list) { + if (opt_inst->info.port == port) { + opt_inst->changed = true; + opt_inst->removed = true; + } + } +} + +static int __team_options_register(struct team *team, + const struct team_option *option, + size_t option_count) +{ + int i; + struct team_option **dst_opts; + int err; + + dst_opts = kzalloc_objs(struct team_option *, option_count); + if (!dst_opts) + return -ENOMEM; + for (i = 0; i < option_count; i++, option++) { + if (__team_find_option(team, option->name)) { + err = -EEXIST; + goto alloc_rollback; + } + dst_opts[i] = kmemdup(option, sizeof(*option), GFP_KERNEL); + if (!dst_opts[i]) { + err = -ENOMEM; + goto alloc_rollback; + } + } + + for (i = 0; i < option_count; i++) { + err = __team_option_inst_add_option(team, dst_opts[i]); + if (err) + goto inst_rollback; + list_add_tail(&dst_opts[i]->list, &team->option_list); + } + + kfree(dst_opts); + return 0; + +inst_rollback: + for (i--; i >= 0; i--) { + __team_option_inst_del_option(team, dst_opts[i]); + list_del(&dst_opts[i]->list); + } + + i = option_count; +alloc_rollback: + for (i--; i >= 0; i--) + kfree(dst_opts[i]); + + kfree(dst_opts); + return err; +} + +static void __team_options_mark_removed(struct team *team, + const struct team_option *option, + size_t option_count) +{ + int i; + + for (i = 0; i < option_count; i++, option++) { + struct team_option *del_opt; + + del_opt = __team_find_option(team, option->name); + if (del_opt) + __team_option_inst_mark_removed_option(team, del_opt); + } +} + +static void __team_options_unregister(struct team *team, + const struct team_option *option, + size_t option_count) +{ + int i; + + for (i = 0; i < option_count; i++, option++) { + struct team_option *del_opt; + + del_opt = __team_find_option(team, option->name); + if (del_opt) { + __team_option_inst_del_option(team, del_opt); + list_del(&del_opt->list); + kfree(del_opt); + } + } +} + +static void __team_options_change_check(struct team *team); + +int team_options_register(struct team *team, + const struct team_option *option, + size_t option_count) +{ + int err; + + err = __team_options_register(team, option, option_count); + if (err) + return err; + __team_options_change_check(team); + return 0; +} +EXPORT_SYMBOL(team_options_register); + +void team_options_unregister(struct team *team, + const struct team_option *option, + size_t option_count) +{ + __team_options_mark_removed(team, option, option_count); + __team_options_change_check(team); + __team_options_unregister(team, option, option_count); +} +EXPORT_SYMBOL(team_options_unregister); + +static int team_option_get(struct team *team, + struct team_option_inst *opt_inst, + struct team_gsetter_ctx *ctx) +{ + if (!opt_inst->option->getter) + return -EOPNOTSUPP; + + opt_inst->option->getter(team, ctx); + return 0; +} + +static int team_option_set(struct team *team, + struct team_option_inst *opt_inst, + struct team_gsetter_ctx *ctx) +{ + if (!opt_inst->option->setter) + return -EOPNOTSUPP; + return opt_inst->option->setter(team, ctx); +} + +void team_option_inst_set_change(struct team_option_inst_info *opt_inst_info) +{ + struct team_option_inst *opt_inst; + + opt_inst = container_of(opt_inst_info, struct team_option_inst, info); + opt_inst->changed = true; +} +EXPORT_SYMBOL(team_option_inst_set_change); + +void team_options_change_check(struct team *team) +{ + __team_options_change_check(team); +} +EXPORT_SYMBOL(team_options_change_check); + + +/**************** + * Mode handling + ****************/ + +static LIST_HEAD(mode_list); +static DEFINE_SPINLOCK(mode_list_lock); + +struct team_mode_item { + struct list_head list; + const struct team_mode *mode; +}; + +static struct team_mode_item *__find_mode(const char *kind) +{ + struct team_mode_item *mitem; + + list_for_each_entry(mitem, &mode_list, list) { + if (strcmp(mitem->mode->kind, kind) == 0) + return mitem; + } + return NULL; +} + +static bool is_good_mode_name(const char *name) +{ + while (*name != '\0') { + if (!isalpha(*name) && !isdigit(*name) && *name != '_') + return false; + name++; + } + return true; +} + +int team_mode_register(const struct team_mode *mode) +{ + int err = 0; + struct team_mode_item *mitem; + + if (!is_good_mode_name(mode->kind) || + mode->priv_size > TEAM_MODE_PRIV_SIZE) + return -EINVAL; + + mitem = kmalloc_obj(*mitem); + if (!mitem) + return -ENOMEM; + + spin_lock(&mode_list_lock); + if (__find_mode(mode->kind)) { + err = -EEXIST; + kfree(mitem); + goto unlock; + } + mitem->mode = mode; + list_add_tail(&mitem->list, &mode_list); +unlock: + spin_unlock(&mode_list_lock); + return err; +} +EXPORT_SYMBOL(team_mode_register); + +void team_mode_unregister(const struct team_mode *mode) +{ + struct team_mode_item *mitem; + + spin_lock(&mode_list_lock); + mitem = __find_mode(mode->kind); + if (mitem) { + list_del_init(&mitem->list); + kfree(mitem); + } + spin_unlock(&mode_list_lock); +} +EXPORT_SYMBOL(team_mode_unregister); + +static const struct team_mode *team_mode_get(const char *kind) +{ + struct team_mode_item *mitem; + const struct team_mode *mode = NULL; + + if (!try_module_get(THIS_MODULE)) + return NULL; + + spin_lock(&mode_list_lock); + mitem = __find_mode(kind); + if (!mitem) { + spin_unlock(&mode_list_lock); + request_module("team-mode-%s", kind); + spin_lock(&mode_list_lock); + mitem = __find_mode(kind); + } + if (mitem) { + mode = mitem->mode; + if (!try_module_get(mode->owner)) + mode = NULL; + } + + spin_unlock(&mode_list_lock); + module_put(THIS_MODULE); + return mode; +} + +static void team_mode_put(const struct team_mode *mode) +{ + module_put(mode->owner); +} + +static bool team_dummy_transmit(struct team *team, struct sk_buff *skb) +{ + dev_kfree_skb_any(skb); + return false; +} + +static rx_handler_result_t team_dummy_receive(struct team *team, + struct team_port *port, + struct sk_buff *skb) +{ + return RX_HANDLER_ANOTHER; +} + +static const struct team_mode __team_no_mode = { + .kind = "*NOMODE*", +}; + +static bool team_is_mode_set(struct team *team) +{ + return team->mode != &__team_no_mode; +} + +static void team_set_no_mode(struct team *team) +{ + team->user_carrier_enabled = false; + team->mode = &__team_no_mode; +} + +static void team_adjust_ops(struct team *team) +{ + /* + * To avoid checks in rx/tx skb paths, ensure here that non-null and + * correct ops are always set. + */ + + if (!team->tx_en_port_count || !team_is_mode_set(team) || + !team->mode->ops->transmit) + WRITE_ONCE(team->ops.transmit, team_dummy_transmit); + else + WRITE_ONCE(team->ops.transmit, team->mode->ops->transmit); + + if (!team->rx_en_port_count || !team_is_mode_set(team) || + !team->mode->ops->receive) + WRITE_ONCE(team->ops.receive, team_dummy_receive); + else + WRITE_ONCE(team->ops.receive, team->mode->ops->receive); +} + +/* + * team_change_mode() ensures no ports are present during mode change, + * but lockless readers can still reach team_xmit(). Avoid touching + * transmit/receive -- they are already set to dummies by + * team_adjust_ops() since no ports are enabled. synchronize_net() + * drains in-flight readers before destroying old mode state. + */ +static int __team_change_mode(struct team *team, + const struct team_mode *new_mode) +{ + /* Check if mode was previously set and do cleanup if so */ + if (team_is_mode_set(team)) { + void (*exit_op)(struct team *team) = team->ops.exit; + + /* Clear cold-path ops used only under RTNL. transmit and + * receive are already dummies (no ports) so leave them + * alone -- overwriting them is the source of the race. + */ + team->ops.init = NULL; + team->ops.exit = NULL; + team->ops.port_enter = NULL; + team->ops.port_leave = NULL; + team->ops.port_change_dev_addr = NULL; + team->ops.port_tx_disabled = NULL; + + /* Wait for in-flight readers before tearing down mode + * state they may reference. + */ + synchronize_net(); + + if (exit_op) + exit_op(team); + team_mode_put(team->mode); + team_set_no_mode(team); + /* zero private data area */ + memset(&team->mode_priv, 0, + sizeof(struct team) - offsetof(struct team, mode_priv)); + } + + if (!new_mode) + return 0; + + if (new_mode->ops->init) { + int err; + + err = new_mode->ops->init(team); + if (err) + return err; + } + + team->mode = new_mode; + team->ops.init = new_mode->ops->init; + team->ops.exit = new_mode->ops->exit; + team->ops.port_enter = new_mode->ops->port_enter; + team->ops.port_leave = new_mode->ops->port_leave; + team->ops.port_change_dev_addr = new_mode->ops->port_change_dev_addr; + team->ops.port_tx_disabled = new_mode->ops->port_tx_disabled; + team_adjust_ops(team); + + return 0; +} + +static int team_change_mode(struct team *team, const char *kind) +{ + const struct team_mode *new_mode; + struct net_device *dev = netdev_from_priv(team); + int err; + + if (!list_empty(&team->port_list)) { + netdev_err(dev, "No ports can be present during mode change\n"); + return -EBUSY; + } + + if (team_is_mode_set(team) && strcmp(team->mode->kind, kind) == 0) { + netdev_err(dev, "Unable to change to the same mode the team is in\n"); + return -EINVAL; + } + + new_mode = team_mode_get(kind); + if (!new_mode) { + netdev_err(dev, "Mode \"%s\" not found\n", kind); + return -EINVAL; + } + + err = __team_change_mode(team, new_mode); + if (err) { + netdev_err(dev, "Failed to change to mode \"%s\"\n", kind); + team_mode_put(new_mode); + return err; + } + + netdev_info(dev, "Mode changed to \"%s\"\n", kind); + return 0; +} + + +/********************* + * Peers notification + *********************/ + +static void team_notify_peers_work(struct work_struct *work) +{ + struct team *team; + int val; + + team = container_of(work, struct team, notify_peers.dw.work); + + if (!rtnl_trylock()) { + schedule_delayed_work(&team->notify_peers.dw, 0); + return; + } + val = atomic_dec_if_positive(&team->notify_peers.count_pending); + if (val < 0) { + rtnl_unlock(); + return; + } + call_netdevice_notifiers(NETDEV_NOTIFY_PEERS, netdev_from_priv(team)); + rtnl_unlock(); + if (val) + schedule_delayed_work(&team->notify_peers.dw, + msecs_to_jiffies(team->notify_peers.interval)); +} + +static void team_notify_peers(struct team *team) +{ + if (!team->notify_peers.count || !netif_running(netdev_from_priv(team))) + return; + atomic_add(team->notify_peers.count, &team->notify_peers.count_pending); + schedule_delayed_work(&team->notify_peers.dw, 0); +} + +static void team_notify_peers_init(struct team *team) +{ + INIT_DELAYED_WORK(&team->notify_peers.dw, team_notify_peers_work); +} + +static void team_notify_peers_fini(struct team *team) +{ + cancel_delayed_work_sync(&team->notify_peers.dw); +} + + +/******************************* + * Send multicast group rejoins + *******************************/ + +static void team_mcast_rejoin_work(struct work_struct *work) +{ + struct team *team; + int val; + + team = container_of(work, struct team, mcast_rejoin.dw.work); + + if (!rtnl_trylock()) { + schedule_delayed_work(&team->mcast_rejoin.dw, 0); + return; + } + val = atomic_dec_if_positive(&team->mcast_rejoin.count_pending); + if (val < 0) { + rtnl_unlock(); + return; + } + call_netdevice_notifiers(NETDEV_RESEND_IGMP, netdev_from_priv(team)); + rtnl_unlock(); + if (val) + schedule_delayed_work(&team->mcast_rejoin.dw, + msecs_to_jiffies(team->mcast_rejoin.interval)); +} + +static void team_mcast_rejoin(struct team *team) +{ + if (!team->mcast_rejoin.count || !netif_running(netdev_from_priv(team))) + return; + atomic_add(team->mcast_rejoin.count, &team->mcast_rejoin.count_pending); + schedule_delayed_work(&team->mcast_rejoin.dw, 0); +} + +static void team_mcast_rejoin_init(struct team *team) +{ + INIT_DELAYED_WORK(&team->mcast_rejoin.dw, team_mcast_rejoin_work); +} + +static void team_mcast_rejoin_fini(struct team *team) +{ + cancel_delayed_work_sync(&team->mcast_rejoin.dw); +} + + +/************************ + * Rx path frame handler + ************************/ + +/* note: already called with rcu_read_lock */ +static rx_handler_result_t team_handle_frame(struct sk_buff **pskb) +{ + struct sk_buff *skb = *pskb; + struct team_port *port; + struct team *team; + rx_handler_result_t res; + + skb = skb_share_check(skb, GFP_ATOMIC); + if (!skb) + return RX_HANDLER_CONSUMED; + + *pskb = skb; + + port = team_port_get_rcu(skb->dev); + team = port->team; + if (!team_port_rx_enabled(port)) { + if (is_link_local_ether_addr(eth_hdr(skb)->h_dest)) + /* link-local packets are mostly useful when stack receives them + * with the link they arrive on. + */ + return RX_HANDLER_PASS; + /* allow exact match delivery for disabled ports */ + res = RX_HANDLER_EXACT; + } else { + res = READ_ONCE(team->ops.receive)(team, port, skb); + } + if (res == RX_HANDLER_ANOTHER) { + struct team_pcpu_stats *pcpu_stats; + + pcpu_stats = this_cpu_ptr(team->pcpu_stats); + u64_stats_update_begin(&pcpu_stats->syncp); + u64_stats_inc(&pcpu_stats->rx_packets); + u64_stats_add(&pcpu_stats->rx_bytes, skb->len); + if (skb->pkt_type == PACKET_MULTICAST) + u64_stats_inc(&pcpu_stats->rx_multicast); + u64_stats_update_end(&pcpu_stats->syncp); + + skb->dev = netdev_from_priv(team); + } else if (res == RX_HANDLER_EXACT) { + this_cpu_inc(team->pcpu_stats->rx_nohandler); + } else { + this_cpu_inc(team->pcpu_stats->rx_dropped); + } + + return res; +} + + +/************************************* + * Multiqueue Tx port select override + *************************************/ + +static int team_queue_override_init(struct team *team) +{ + struct list_head *listarr; + unsigned int queue_cnt = netdev_from_priv(team)->num_tx_queues - 1; + unsigned int i; + + if (!queue_cnt) + return 0; + listarr = kmalloc_objs(struct list_head, queue_cnt); + if (!listarr) + return -ENOMEM; + team->qom_lists = listarr; + for (i = 0; i < queue_cnt; i++) + INIT_LIST_HEAD(listarr++); + return 0; +} + +static void team_queue_override_fini(struct team *team) +{ + kfree(team->qom_lists); +} + +static struct list_head *__team_get_qom_list(struct team *team, u16 queue_id) +{ + return &team->qom_lists[queue_id - 1]; +} + +/* + * note: already called with rcu_read_lock + */ +static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb) +{ + struct list_head *qom_list; + struct team_port *port; + + if (!team->queue_override_enabled || !skb->queue_mapping) + return false; + qom_list = __team_get_qom_list(team, skb->queue_mapping); + list_for_each_entry_rcu(port, qom_list, qom_list) { + if (!team_dev_queue_xmit(team, port, skb)) + return true; + } + return false; +} + +static void __team_queue_override_port_del(struct team *team, + struct team_port *port) +{ + if (!port->queue_id) + return; + list_del_rcu(&port->qom_list); +} + +static bool team_queue_override_port_has_gt_prio_than(struct team_port *port, + struct team_port *cur) +{ + if (port->priority < cur->priority) + return true; + if (port->priority > cur->priority) + return false; + if (port->tx_index < cur->tx_index) + return true; + return false; +} + +static void __team_queue_override_port_add(struct team *team, + struct team_port *port) +{ + struct team_port *cur; + struct list_head *qom_list; + struct list_head *node; + + if (!port->queue_id) + return; + qom_list = __team_get_qom_list(team, port->queue_id); + node = qom_list; + list_for_each_entry(cur, qom_list, qom_list) { + if (team_queue_override_port_has_gt_prio_than(port, cur)) + break; + node = &cur->qom_list; + } + list_add_tail_rcu(&port->qom_list, node); +} + +static void __team_queue_override_enabled_check(struct team *team) +{ + struct team_port *port; + bool enabled = false; + + list_for_each_entry(port, &team->port_list, list) { + if (port->queue_id) { + enabled = true; + break; + } + } + if (enabled == team->queue_override_enabled) + return; + netdev_dbg(netdev_from_priv(team), "%s queue override\n", + enabled ? "Enabling" : "Disabling"); + team->queue_override_enabled = enabled; +} + +static void team_queue_override_port_prio_changed(struct team *team, + struct team_port *port) +{ + if (!port->queue_id || !team_port_tx_enabled(port)) + return; + __team_queue_override_port_del(team, port); + __team_queue_override_port_add(team, port); + __team_queue_override_enabled_check(team); +} + +static void team_queue_override_port_change_queue_id(struct team *team, + struct team_port *port, + u16 new_queue_id) +{ + if (team_port_tx_enabled(port)) { + __team_queue_override_port_del(team, port); + port->queue_id = new_queue_id; + __team_queue_override_port_add(team, port); + __team_queue_override_enabled_check(team); + } else { + port->queue_id = new_queue_id; + } +} + +static void team_queue_override_port_add(struct team *team, + struct team_port *port) +{ + __team_queue_override_port_add(team, port); + __team_queue_override_enabled_check(team); +} + +static void team_queue_override_port_del(struct team *team, + struct team_port *port) +{ + __team_queue_override_port_del(team, port); + __team_queue_override_enabled_check(team); +} + + +/**************** + * Port handling + ****************/ + +static bool team_port_find(const struct team *team, + const struct team_port *port) +{ + struct team_port *cur; + + list_for_each_entry(cur, &team->port_list, list) + if (cur == port) + return true; + return false; +} + +static void __team_port_enable_rx(struct team *team, + struct team_port *port) +{ + team->rx_en_port_count++; + WRITE_ONCE(port->rx_enabled, true); +} + +static void __team_port_disable_rx(struct team *team, + struct team_port *port) +{ + team->rx_en_port_count--; + WRITE_ONCE(port->rx_enabled, false); +} + +static void team_port_enable_rx(struct team *team, + struct team_port *port) +{ + if (team_port_rx_enabled(port)) + return; + + __team_port_enable_rx(team, port); + team_adjust_ops(team); + team_notify_peers(team); + team_mcast_rejoin(team); +} + +static void team_port_disable_rx(struct team *team, + struct team_port *port) +{ + if (!team_port_rx_enabled(port)) + return; + + __team_port_disable_rx(team, port); + team_adjust_ops(team); +} + +/* + * Enable just TX on the port by adding to tx-enabled port hashlist and + * setting port->tx_index (Might be racy so reader could see incorrect + * ifindex when processing a flying packet, but that is not a problem). + * Write guarded by RTNL. + */ +static void __team_port_enable_tx(struct team *team, + struct team_port *port) +{ + WRITE_ONCE(port->tx_index, team->tx_en_port_count); + WRITE_ONCE(team->tx_en_port_count, team->tx_en_port_count + 1); + hlist_add_head_rcu(&port->tx_hlist, + team_tx_port_index_hash(team, port->tx_index)); +} + +static void team_port_enable_tx(struct team *team, + struct team_port *port) +{ + if (team_port_tx_enabled(port)) + return; + + __team_port_enable_tx(team, port); + team_adjust_ops(team); + team_queue_override_port_add(team, port); + + /* Don't rejoin multicast, since this port might not be receiving. */ + team_notify_peers(team); + team_lower_state_changed(port); +} + +static void __reconstruct_port_hlist(struct team *team, int rm_index) +{ + struct hlist_head *tx_port_index_hash; + struct team_port *port; + int i; + + for (i = rm_index + 1; i < team->tx_en_port_count; i++) { + port = team_get_port_by_tx_index(team, i); + hlist_del_rcu(&port->tx_hlist); + WRITE_ONCE(port->tx_index, port->tx_index - 1); + tx_port_index_hash = team_tx_port_index_hash(team, + port->tx_index); + hlist_add_head_rcu(&port->tx_hlist, tx_port_index_hash); + } +} + +static void __team_port_disable_tx(struct team *team, + struct team_port *port) +{ + if (team->ops.port_tx_disabled) + team->ops.port_tx_disabled(team, port); + + hlist_del_rcu(&port->tx_hlist); + __reconstruct_port_hlist(team, port->tx_index); + + WRITE_ONCE(port->tx_index, -1); + WRITE_ONCE(team->tx_en_port_count, team->tx_en_port_count - 1); +} + +static void team_port_disable_tx(struct team *team, + struct team_port *port) +{ + if (!team_port_tx_enabled(port)) + return; + + __team_port_disable_tx(team, port); + + team_queue_override_port_del(team, port); + team_adjust_ops(team); + team_lower_state_changed(port); +} + +/* + * Enable TX AND RX on the port. + */ +static void team_port_enable(struct team *team, + struct team_port *port) +{ + bool rx_was_enabled; + bool tx_was_enabled; + + if (team_port_enabled(port)) + return; + + rx_was_enabled = team_port_rx_enabled(port); + tx_was_enabled = team_port_tx_enabled(port); + + if (!rx_was_enabled) + __team_port_enable_rx(team, port); + if (!tx_was_enabled) + __team_port_enable_tx(team, port); + + team_adjust_ops(team); + if (!tx_was_enabled) + team_queue_override_port_add(team, port); + team_notify_peers(team); + if (!rx_was_enabled) + team_mcast_rejoin(team); + if (!tx_was_enabled) + team_lower_state_changed(port); +} + +static void team_port_disable(struct team *team, + struct team_port *port) +{ + bool rx_was_enabled = team_port_rx_enabled(port); + bool tx_was_enabled = team_port_tx_enabled(port); + + if (!tx_was_enabled && !rx_was_enabled) + return; + + if (tx_was_enabled) { + __team_port_disable_tx(team, port); + team_queue_override_port_del(team, port); + } + if (rx_was_enabled) + __team_port_disable_rx(team, port); + + team_adjust_ops(team); + + if (tx_was_enabled) + team_lower_state_changed(port); +} + +static int team_port_enter(struct team *team, struct team_port *port) +{ + int err = 0; + + dev_hold(netdev_from_priv(team)); + if (team->ops.port_enter) { + err = team->ops.port_enter(team, port); + if (err) { + netdev_err(netdev_from_priv(team), + "Device %s failed to enter team mode\n", + port->dev->name); + goto err_port_enter; + } + } + + return 0; + +err_port_enter: + dev_put(netdev_from_priv(team)); + + return err; +} + +static void team_port_leave(struct team *team, struct team_port *port) +{ + if (team->ops.port_leave) + team->ops.port_leave(team, port); + dev_put(netdev_from_priv(team)); +} + +#ifdef CONFIG_NET_POLL_CONTROLLER +static int __team_port_enable_netpoll(struct team_port *port) +{ + struct netpoll *np; + int err; + + np = kzalloc_obj(*np); + if (!np) + return -ENOMEM; + + err = __netpoll_setup(np, port->dev); + if (err) { + kfree(np); + return err; + } + port->np = np; + return err; +} + +static int team_port_enable_netpoll(struct team_port *port) +{ + if (!netdev_from_priv(port->team)->npinfo) + return 0; + + return __team_port_enable_netpoll(port); +} + +static void team_port_disable_netpoll(struct team_port *port) +{ + struct netpoll *np = port->np; + + if (!np) + return; + port->np = NULL; + + __netpoll_free(np); +} +#else +static int team_port_enable_netpoll(struct team_port *port) +{ + return 0; +} +static void team_port_disable_netpoll(struct team_port *port) +{ +} +#endif + +static int team_upper_dev_link(struct team *team, struct team_port *port, + struct netlink_ext_ack *extack) +{ + struct netdev_lag_upper_info lag_upper_info; + int err; + + lag_upper_info.tx_type = team->mode->lag_tx_type; + lag_upper_info.hash_type = NETDEV_LAG_HASH_UNKNOWN; + err = netdev_master_upper_dev_link(port->dev, netdev_from_priv(team), + NULL, &lag_upper_info, extack); + if (err) + return err; + port->dev->priv_flags |= IFF_TEAM_PORT; + return 0; +} + +static void team_upper_dev_unlink(struct team *team, struct team_port *port) +{ + netdev_upper_dev_unlink(port->dev, netdev_from_priv(team)); + port->dev->priv_flags &= ~IFF_TEAM_PORT; +} + +static void __team_port_change_port_added(struct team_port *port, bool linkup); +static int team_dev_type_check_change(struct net_device *dev, + struct net_device *port_dev); + +static int team_port_add(struct team *team, struct net_device *port_dev, + struct netlink_ext_ack *extack) +{ + struct net_device *dev = netdev_from_priv(team); + struct team_port *port; + char *portname = port_dev->name; + int err; + + if (port_dev->flags & IFF_LOOPBACK) { + NL_SET_ERR_MSG(extack, "Loopback device can't be added as a team port"); + netdev_err(dev, "Device %s is loopback device. Loopback devices can't be added as a team port\n", + portname); + return -EINVAL; + } + + if (netif_is_team_port(port_dev)) { + NL_SET_ERR_MSG(extack, "Device is already a port of a team device"); + netdev_err(dev, "Device %s is already a port " + "of a team device\n", portname); + return -EBUSY; + } + + if (dev == port_dev) { + NL_SET_ERR_MSG(extack, "Cannot enslave team device to itself"); + netdev_err(dev, "Cannot enslave team device to itself\n"); + return -EINVAL; + } + + if (netdev_has_upper_dev(dev, port_dev)) { + NL_SET_ERR_MSG(extack, "Device is already an upper device of the team interface"); + netdev_err(dev, "Device %s is already an upper device of the team interface\n", + portname); + return -EBUSY; + } + + if (netdev_has_upper_dev(port_dev, dev)) { + NL_SET_ERR_MSG(extack, "Device is already a lower device of the team interface"); + netdev_err(dev, "Device %s is already a lower device of the team interface\n", + portname); + return -EBUSY; + } + + if (port_dev->features & NETIF_F_VLAN_CHALLENGED && + vlan_uses_dev(dev)) { + NL_SET_ERR_MSG(extack, "Device is VLAN challenged and team device has VLAN set up"); + netdev_err(dev, "Device %s is VLAN challenged and team device has VLAN set up\n", + portname); + return -EPERM; + } + + if (port_dev->flags & IFF_UP) { + NL_SET_ERR_MSG(extack, "Device is up. Set it down before adding it as a team port"); + netdev_err(dev, "Device %s is up. Set it down before adding it as a team port\n", + portname); + return -EBUSY; + } + + port = kzalloc(sizeof(struct team_port) + team->mode->port_priv_size, + GFP_KERNEL); + if (!port) + return -ENOMEM; + + port->dev = port_dev; + port->team = team; + INIT_LIST_HEAD(&port->qom_list); + + port->orig.mtu = port_dev->mtu; + /* + * MTU assignment will be handled in team_dev_type_check_change + * if dev and port_dev are of different types + */ + if (dev->type == port_dev->type) { + err = dev_set_mtu(port_dev, dev->mtu); + if (err) { + netdev_dbg(dev, "Error %d calling dev_set_mtu\n", err); + goto err_set_mtu; + } + } + + memcpy(port->orig.dev_addr, port_dev->dev_addr, port_dev->addr_len); + + err = team_port_enter(team, port); + if (err) { + netdev_err(dev, "Device %s failed to enter team mode\n", + portname); + goto err_port_enter; + } + + err = dev_open(port_dev, extack); + if (err) { + netdev_dbg(dev, "Device %s opening failed\n", + portname); + goto err_dev_open; + } + + err = vlan_vids_add_by_dev(port_dev, dev); + if (err) { + netdev_err(dev, "Failed to add vlan ids to device %s\n", + portname); + goto err_vids_add; + } + + err = team_port_enable_netpoll(port); + if (err) { + netdev_err(dev, "Failed to enable netpoll on device %s\n", + portname); + goto err_enable_netpoll; + } + + if (!(dev->features & NETIF_F_LRO)) + dev_disable_lro(port_dev); + + err = netdev_rx_handler_register(port_dev, team_handle_frame, + port); + if (err) { + netdev_err(dev, "Device %s failed to register rx_handler\n", + portname); + goto err_handler_register; + } + + err = team_upper_dev_link(team, port, extack); + if (err) { + netdev_err(dev, "Device %s failed to set upper link\n", + portname); + goto err_set_upper_link; + } + + err = __team_option_inst_add_port(team, port); + if (err) { + netdev_err(dev, "Device %s failed to add per-port options\n", + portname); + goto err_option_port_add; + } + + /* set promiscuity level to new slave */ + if (dev->flags & IFF_PROMISC) { + err = dev_set_promiscuity(port_dev, 1); + if (err) + goto err_set_slave_promisc; + } + + /* set allmulti level to new slave */ + if (dev->flags & IFF_ALLMULTI) { + err = dev_set_allmulti(port_dev, 1); + if (err) { + if (dev->flags & IFF_PROMISC) + dev_set_promiscuity(port_dev, -1); + goto err_set_slave_allmulti; + } + } + + err = team_dev_type_check_change(dev, port_dev); + if (err) + goto err_set_dev_type; + + if (dev->flags & IFF_UP) { + netif_addr_lock_bh(dev); + dev_uc_sync_multiple(port_dev, dev); + dev_mc_sync_multiple(port_dev, dev); + netif_addr_unlock_bh(dev); + } + + WRITE_ONCE(port->tx_index, -1); + list_add_tail_rcu(&port->list, &team->port_list); + team_port_enable(team, port); + netdev_compute_master_upper_features(dev, true); + netdev_lock_ops(port_dev); + __team_port_change_port_added(port, !!netif_oper_up(port_dev)); + netdev_unlock_ops(port_dev); + __team_options_change_check(team); + + netdev_info(dev, "Port device %s added\n", portname); + + return 0; + +err_set_dev_type: +err_set_slave_allmulti: +err_set_slave_promisc: + __team_option_inst_del_port(team, port); + +err_option_port_add: + team_upper_dev_unlink(team, port); + +err_set_upper_link: + netdev_rx_handler_unregister(port_dev); + +err_handler_register: + team_port_disable_netpoll(port); + +err_enable_netpoll: + vlan_vids_del_by_dev(port_dev, dev); + +err_vids_add: + dev_close(port_dev); + +err_dev_open: + team_port_leave(team, port); + team_port_set_orig_dev_addr(port); + +err_port_enter: + dev_set_mtu(port_dev, port->orig.mtu); + +err_set_mtu: + kfree(port); + + return err; +} + +static void __team_port_change_port_removed(struct team_port *port); + +static int team_port_del(struct team *team, struct net_device *port_dev, bool unregister) +{ + struct net_device *dev = netdev_from_priv(team); + struct team_port *port; + char *portname = port_dev->name; + + port = team_port_get_rtnl(port_dev); + if (!port || !team_port_find(team, port)) { + netdev_err(dev, "Device %s does not act as a port of this team\n", + portname); + return -ENOENT; + } + + team_port_disable(team, port); + list_del_rcu(&port->list); + + if (dev->flags & IFF_PROMISC) + dev_set_promiscuity(port_dev, -1); + if (dev->flags & IFF_ALLMULTI) + dev_set_allmulti(port_dev, -1); + + team_upper_dev_unlink(team, port); + netdev_rx_handler_unregister(port_dev); + team_port_disable_netpoll(port); + vlan_vids_del_by_dev(port_dev, dev); + if (dev->flags & IFF_UP) { + dev_uc_unsync(port_dev, dev); + dev_mc_unsync(port_dev, dev); + } + dev_close(port_dev); + team_port_leave(team, port); + + __team_option_inst_mark_removed_port(team, port); + __team_options_change_check(team); + __team_option_inst_del_port(team, port); + __team_port_change_port_removed(port); + + team_port_set_orig_dev_addr(port); + if (unregister) { + netdev_lock_ops(port_dev); + __netif_set_mtu(port_dev, port->orig.mtu); + netdev_unlock_ops(port_dev); + } else { + dev_set_mtu(port_dev, port->orig.mtu); + } + kfree_rcu(port, rcu); + netdev_info(dev, "Port device %s removed\n", portname); + netdev_compute_master_upper_features(dev, true); + + return 0; +} + + +/***************** + * Net device ops + *****************/ + +static void team_mode_option_get(struct team *team, struct team_gsetter_ctx *ctx) +{ + ctx->data.str_val = team->mode->kind; +} + +static int team_mode_option_set(struct team *team, struct team_gsetter_ctx *ctx) +{ + return team_change_mode(team, ctx->data.str_val); +} + +static void team_notify_peers_count_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + ctx->data.u32_val = team->notify_peers.count; +} + +static int team_notify_peers_count_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + team->notify_peers.count = ctx->data.u32_val; + return 0; +} + +static void team_notify_peers_interval_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + ctx->data.u32_val = team->notify_peers.interval; +} + +static int team_notify_peers_interval_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + team->notify_peers.interval = ctx->data.u32_val; + return 0; +} + +static void team_mcast_rejoin_count_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + ctx->data.u32_val = team->mcast_rejoin.count; +} + +static int team_mcast_rejoin_count_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + team->mcast_rejoin.count = ctx->data.u32_val; + return 0; +} + +static void team_mcast_rejoin_interval_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + ctx->data.u32_val = team->mcast_rejoin.interval; +} + +static int team_mcast_rejoin_interval_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + team->mcast_rejoin.interval = ctx->data.u32_val; + return 0; +} + +static void team_port_en_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.bool_val = team_port_enabled(port); +} + +static int team_port_en_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + if (ctx->data.bool_val) + team_port_enable(team, port); + else + team_port_disable(team, port); + return 0; +} + +static void team_port_rx_en_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.bool_val = team_port_rx_enabled(port); +} + +static int team_port_rx_en_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + if (ctx->data.bool_val) + team_port_enable_rx(team, port); + else + team_port_disable_rx(team, port); + return 0; +} + +static void team_port_tx_en_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.bool_val = team_port_tx_enabled(port); +} + +static int team_port_tx_en_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + if (ctx->data.bool_val) + team_port_enable_tx(team, port); + else + team_port_disable_tx(team, port); + return 0; +} + +static void team_user_linkup_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.bool_val = port->user.linkup; +} + +static void __team_carrier_check(struct team *team); + +static int team_user_linkup_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + port->user.linkup = ctx->data.bool_val; + team_refresh_port_linkup(port); + __team_carrier_check(port->team); + return 0; +} + +static void team_user_linkup_en_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.bool_val = port->user.linkup_enabled; +} + +static int team_user_linkup_en_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + port->user.linkup_enabled = ctx->data.bool_val; + team_refresh_port_linkup(port); + __team_carrier_check(port->team); + return 0; +} + +static void team_priority_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.s32_val = port->priority; +} + +static int team_priority_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + s32 priority = ctx->data.s32_val; + + if (port->priority == priority) + return 0; + port->priority = priority; + team_queue_override_port_prio_changed(team, port); + return 0; +} + +static void team_queue_id_option_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + + ctx->data.u32_val = port->queue_id; +} + +static int team_queue_id_option_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + u16 new_queue_id = ctx->data.u32_val; + + if (port->queue_id == new_queue_id) + return 0; + if (new_queue_id >= netdev_from_priv(team)->real_num_tx_queues) + return -EINVAL; + team_queue_override_port_change_queue_id(team, port, new_queue_id); + return 0; +} + +static const struct team_option team_options[] = { + { + .name = "mode", + .type = TEAM_OPTION_TYPE_STRING, + .getter = team_mode_option_get, + .setter = team_mode_option_set, + }, + { + .name = "notify_peers_count", + .type = TEAM_OPTION_TYPE_U32, + .getter = team_notify_peers_count_get, + .setter = team_notify_peers_count_set, + }, + { + .name = "notify_peers_interval", + .type = TEAM_OPTION_TYPE_U32, + .getter = team_notify_peers_interval_get, + .setter = team_notify_peers_interval_set, + }, + { + .name = "mcast_rejoin_count", + .type = TEAM_OPTION_TYPE_U32, + .getter = team_mcast_rejoin_count_get, + .setter = team_mcast_rejoin_count_set, + }, + { + .name = "mcast_rejoin_interval", + .type = TEAM_OPTION_TYPE_U32, + .getter = team_mcast_rejoin_interval_get, + .setter = team_mcast_rejoin_interval_set, + }, + { + .name = "enabled", + .type = TEAM_OPTION_TYPE_BOOL, + .per_port = true, + .getter = team_port_en_option_get, + .setter = team_port_en_option_set, + }, + { + .name = "rx_enabled", + .type = TEAM_OPTION_TYPE_BOOL, + .per_port = true, + .getter = team_port_rx_en_option_get, + .setter = team_port_rx_en_option_set, + }, + { + .name = "tx_enabled", + .type = TEAM_OPTION_TYPE_BOOL, + .per_port = true, + .getter = team_port_tx_en_option_get, + .setter = team_port_tx_en_option_set, + }, + { + .name = "user_linkup", + .type = TEAM_OPTION_TYPE_BOOL, + .per_port = true, + .getter = team_user_linkup_option_get, + .setter = team_user_linkup_option_set, + }, + { + .name = "user_linkup_enabled", + .type = TEAM_OPTION_TYPE_BOOL, + .per_port = true, + .getter = team_user_linkup_en_option_get, + .setter = team_user_linkup_en_option_set, + }, + { + .name = "priority", + .type = TEAM_OPTION_TYPE_S32, + .per_port = true, + .getter = team_priority_option_get, + .setter = team_priority_option_set, + }, + { + .name = "queue_id", + .type = TEAM_OPTION_TYPE_U32, + .per_port = true, + .getter = team_queue_id_option_get, + .setter = team_queue_id_option_set, + }, +}; + + +static int team_init(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + int i; + int err; + + team_set_no_mode(team); + team->notifier_ctx = false; + + team->pcpu_stats = netdev_alloc_pcpu_stats(struct team_pcpu_stats); + if (!team->pcpu_stats) + return -ENOMEM; + + for (i = 0; i < TEAM_PORT_HASHENTRIES; i++) + INIT_HLIST_HEAD(&team->tx_en_port_hlist[i]); + INIT_LIST_HEAD(&team->port_list); + err = team_queue_override_init(team); + if (err) + goto err_team_queue_override_init; + + team_adjust_ops(team); + + INIT_LIST_HEAD(&team->option_list); + INIT_LIST_HEAD(&team->option_inst_list); + + team_notify_peers_init(team); + team_mcast_rejoin_init(team); + + err = team_options_register(team, team_options, ARRAY_SIZE(team_options)); + if (err) + goto err_options_register; + netif_carrier_off(dev); + + netdev_lockdep_set_classes(dev); + + return 0; + +err_options_register: + team_mcast_rejoin_fini(team); + team_notify_peers_fini(team); + team_queue_override_fini(team); +err_team_queue_override_init: + free_percpu(team->pcpu_stats); + + return err; +} + +static void team_uninit(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + struct team_port *tmp; + + ASSERT_RTNL(); + + list_for_each_entry_safe(port, tmp, &team->port_list, list) + team_port_del(team, port->dev, false); + + __team_change_mode(team, NULL); /* cleanup */ + __team_options_unregister(team, team_options, ARRAY_SIZE(team_options)); + team_mcast_rejoin_fini(team); + team_notify_peers_fini(team); + team_queue_override_fini(team); + netdev_change_features(dev); +} + +static void team_destructor(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + + free_percpu(team->pcpu_stats); +} + +static int team_open(struct net_device *dev) +{ + return 0; +} + +static int team_close(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + + list_for_each_entry(port, &team->port_list, list) { + dev_uc_unsync(port->dev, dev); + dev_mc_unsync(port->dev, dev); + } + + return 0; +} + +/* + * note: already called with rcu_read_lock + */ +static netdev_tx_t team_xmit(struct sk_buff *skb, struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + bool tx_success; + unsigned int len = skb->len; + + tx_success = team_queue_override_transmit(team, skb); + if (!tx_success) + tx_success = READ_ONCE(team->ops.transmit)(team, skb); + if (tx_success) { + struct team_pcpu_stats *pcpu_stats; + + pcpu_stats = this_cpu_ptr(team->pcpu_stats); + u64_stats_update_begin(&pcpu_stats->syncp); + u64_stats_inc(&pcpu_stats->tx_packets); + u64_stats_add(&pcpu_stats->tx_bytes, len); + u64_stats_update_end(&pcpu_stats->syncp); + } else { + this_cpu_inc(team->pcpu_stats->tx_dropped); + } + + return NETDEV_TX_OK; +} + +static u16 team_select_queue(struct net_device *dev, struct sk_buff *skb, + struct net_device *sb_dev) +{ + /* + * This helper function exists to help dev_pick_tx get the correct + * destination queue. Using a helper function skips a call to + * skb_tx_hash and will put the skbs in the queue we expect on their + * way down to the team driver. + */ + u16 txq = skb_rx_queue_recorded(skb) ? skb_get_rx_queue(skb) : 0; + + /* + * Save the original txq to restore before passing to the driver + */ + qdisc_skb_cb(skb)->slave_dev_queue_mapping = skb->queue_mapping; + + if (unlikely(txq >= dev->real_num_tx_queues)) { + do { + txq -= dev->real_num_tx_queues; + } while (txq >= dev->real_num_tx_queues); + } + return txq; +} + +static void team_change_rx_flags(struct net_device *dev, int change) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + int inc; + + ASSERT_RTNL(); + + list_for_each_entry(port, &team->port_list, list) { + if (change & IFF_PROMISC) { + inc = dev->flags & IFF_PROMISC ? 1 : -1; + dev_set_promiscuity(port->dev, inc); + } + if (change & IFF_ALLMULTI) { + inc = dev->flags & IFF_ALLMULTI ? 1 : -1; + dev_set_allmulti(port->dev, inc); + } + } +} + +static void team_set_rx_mode(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + + rcu_read_lock(); + list_for_each_entry_rcu(port, &team->port_list, list) { + dev_uc_sync_multiple(port->dev, dev); + dev_mc_sync_multiple(port->dev, dev); + } + rcu_read_unlock(); +} + +static int team_set_mac_address(struct net_device *dev, void *p) +{ + struct sockaddr *addr = p; + struct team *team = netdev_priv(dev); + struct team_port *port; + + ASSERT_RTNL(); + + if (dev->type == ARPHRD_ETHER && !is_valid_ether_addr(addr->sa_data)) + return -EADDRNOTAVAIL; + dev_addr_set(dev, addr->sa_data); + list_for_each_entry(port, &team->port_list, list) + if (team->ops.port_change_dev_addr) + team->ops.port_change_dev_addr(team, port); + return 0; +} + +static int team_change_mtu(struct net_device *dev, int new_mtu) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + int err; + + ASSERT_RTNL(); + + team->port_mtu_change_allowed = true; + list_for_each_entry(port, &team->port_list, list) { + err = dev_set_mtu(port->dev, new_mtu); + if (err) { + netdev_err(dev, "Device %s failed to change mtu", + port->dev->name); + goto unwind; + } + } + team->port_mtu_change_allowed = false; + + WRITE_ONCE(dev->mtu, new_mtu); + + return 0; + +unwind: + list_for_each_entry_continue_reverse(port, &team->port_list, list) + dev_set_mtu(port->dev, dev->mtu); + team->port_mtu_change_allowed = false; + + return err; +} + +static void +team_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats) +{ + struct team *team = netdev_priv(dev); + struct team_pcpu_stats *p; + u64 rx_packets, rx_bytes, rx_multicast, tx_packets, tx_bytes; + u32 rx_dropped = 0, tx_dropped = 0, rx_nohandler = 0; + unsigned int start; + int i; + + for_each_possible_cpu(i) { + p = per_cpu_ptr(team->pcpu_stats, i); + do { + start = u64_stats_fetch_begin(&p->syncp); + rx_packets = u64_stats_read(&p->rx_packets); + rx_bytes = u64_stats_read(&p->rx_bytes); + rx_multicast = u64_stats_read(&p->rx_multicast); + tx_packets = u64_stats_read(&p->tx_packets); + tx_bytes = u64_stats_read(&p->tx_bytes); + } while (u64_stats_fetch_retry(&p->syncp, start)); + + stats->rx_packets += rx_packets; + stats->rx_bytes += rx_bytes; + stats->multicast += rx_multicast; + stats->tx_packets += tx_packets; + stats->tx_bytes += tx_bytes; + /* + * rx_dropped, tx_dropped & rx_nohandler are u32, + * updated without syncp protection. + */ + rx_dropped += READ_ONCE(p->rx_dropped); + tx_dropped += READ_ONCE(p->tx_dropped); + rx_nohandler += READ_ONCE(p->rx_nohandler); + } + stats->rx_dropped = rx_dropped; + stats->tx_dropped = tx_dropped; + stats->rx_nohandler = rx_nohandler; +} + +static int team_vlan_rx_add_vid(struct net_device *dev, __be16 proto, u16 vid) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + int err; + + ASSERT_RTNL(); + + list_for_each_entry(port, &team->port_list, list) { + err = vlan_vid_add(port->dev, proto, vid); + if (err) + goto unwind; + } + + return 0; + +unwind: + list_for_each_entry_continue_reverse(port, &team->port_list, list) + vlan_vid_del(port->dev, proto, vid); + + return err; +} + +static int team_vlan_rx_kill_vid(struct net_device *dev, __be16 proto, u16 vid) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + + ASSERT_RTNL(); + + list_for_each_entry(port, &team->port_list, list) + vlan_vid_del(port->dev, proto, vid); + + return 0; +} + +#ifdef CONFIG_NET_POLL_CONTROLLER +static void team_poll_controller(struct net_device *dev) +{ +} + +static void __team_netpoll_cleanup(struct team *team) +{ + struct team_port *port; + + list_for_each_entry(port, &team->port_list, list) + team_port_disable_netpoll(port); +} + +static void team_netpoll_cleanup(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + + ASSERT_RTNL(); + + __team_netpoll_cleanup(team); +} + +static int team_netpoll_setup(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + struct team_port *port; + int err = 0; + + ASSERT_RTNL(); + + list_for_each_entry(port, &team->port_list, list) { + err = __team_port_enable_netpoll(port); + if (err) { + __team_netpoll_cleanup(team); + break; + } + } + return err; +} +#endif + +static int team_add_slave(struct net_device *dev, struct net_device *port_dev, + struct netlink_ext_ack *extack) +{ + struct team *team = netdev_priv(dev); + + ASSERT_RTNL(); + + return team_port_add(team, port_dev, extack); +} + +static int team_del_slave(struct net_device *dev, struct net_device *port_dev) +{ + struct team *team = netdev_priv(dev); + + ASSERT_RTNL(); + + return team_port_del(team, port_dev, false); +} + +static int team_del_slave_on_unregister(struct net_device *dev, struct net_device *port_dev) +{ + struct team *team = netdev_priv(dev); + + ASSERT_RTNL(); + + return team_port_del(team, port_dev, true); +} + +static netdev_features_t team_fix_features(struct net_device *dev, + netdev_features_t features) +{ + struct team_port *port; + struct team *team = netdev_priv(dev); + netdev_features_t mask; + + mask = features; + features = netdev_base_features(features); + + rcu_read_lock(); + list_for_each_entry_rcu(port, &team->port_list, list) { + features = netdev_increment_features(features, + port->dev->features, + mask); + } + rcu_read_unlock(); + + features = netdev_add_tso_features(features, mask); + + return features; +} + +static int team_change_carrier(struct net_device *dev, bool new_carrier) +{ + struct team *team = netdev_priv(dev); + + team->user_carrier_enabled = true; + + if (new_carrier) + netif_carrier_on(dev); + else + netif_carrier_off(dev); + return 0; +} + +static const struct net_device_ops team_netdev_ops = { + .ndo_init = team_init, + .ndo_uninit = team_uninit, + .ndo_open = team_open, + .ndo_stop = team_close, + .ndo_start_xmit = team_xmit, + .ndo_select_queue = team_select_queue, + .ndo_change_rx_flags = team_change_rx_flags, + .ndo_set_rx_mode = team_set_rx_mode, + .ndo_set_mac_address = team_set_mac_address, + .ndo_change_mtu = team_change_mtu, + .ndo_get_stats64 = team_get_stats64, + .ndo_vlan_rx_add_vid = team_vlan_rx_add_vid, + .ndo_vlan_rx_kill_vid = team_vlan_rx_kill_vid, +#ifdef CONFIG_NET_POLL_CONTROLLER + .ndo_poll_controller = team_poll_controller, + .ndo_netpoll_setup = team_netpoll_setup, + .ndo_netpoll_cleanup = team_netpoll_cleanup, +#endif + .ndo_add_slave = team_add_slave, + .ndo_del_slave = team_del_slave, + .ndo_fix_features = team_fix_features, + .ndo_change_carrier = team_change_carrier, + .ndo_features_check = passthru_features_check, +}; + +/*********************** + * ethtool interface + ***********************/ + +static void team_ethtool_get_drvinfo(struct net_device *dev, + struct ethtool_drvinfo *drvinfo) +{ + strscpy(drvinfo->driver, DRV_NAME, sizeof(drvinfo->driver)); +} + +static int team_ethtool_get_link_ksettings(struct net_device *dev, + struct ethtool_link_ksettings *cmd) +{ + struct team *team= netdev_priv(dev); + unsigned long speed = 0; + struct team_port *port; + + cmd->base.duplex = DUPLEX_UNKNOWN; + cmd->base.port = PORT_OTHER; + + rcu_read_lock(); + list_for_each_entry_rcu(port, &team->port_list, list) { + if (team_port_txable(port)) { + if (port->state.speed != SPEED_UNKNOWN) + speed += port->state.speed; + if (cmd->base.duplex == DUPLEX_UNKNOWN && + port->state.duplex != DUPLEX_UNKNOWN) + cmd->base.duplex = port->state.duplex; + } + } + rcu_read_unlock(); + + cmd->base.speed = speed ? : SPEED_UNKNOWN; + + return 0; +} + +static const struct ethtool_ops team_ethtool_ops = { + .get_drvinfo = team_ethtool_get_drvinfo, + .get_link = ethtool_op_get_link, + .get_link_ksettings = team_ethtool_get_link_ksettings, +}; + +/*********************** + * rt netlink interface + ***********************/ + +/* For tx path we need a linkup && enabled port and for parse any port + * suffices. + */ +static struct team_port *team_header_port_get_rcu(struct team *team, + bool txable) +{ + struct team_port *port; + + list_for_each_entry_rcu(port, &team->port_list, list) { + if (!txable || team_port_txable(port)) + return port; + } + + return NULL; +} + +static int team_header_create(struct sk_buff *skb, struct net_device *team_dev, + unsigned short type, const void *daddr, + const void *saddr, unsigned int len) +{ + struct team *team = netdev_priv(team_dev); + const struct header_ops *port_ops; + struct team_port *port; + int ret = 0; + + rcu_read_lock(); + port = team_header_port_get_rcu(team, true); + if (port) { + port_ops = READ_ONCE(port->dev->header_ops); + if (port_ops && port_ops->create) + ret = port_ops->create(skb, port->dev, + type, daddr, saddr, len); + } + rcu_read_unlock(); + return ret; +} + +static int team_header_parse(const struct sk_buff *skb, + const struct net_device *team_dev, + unsigned char *haddr) +{ + struct team *team = netdev_priv(team_dev); + const struct header_ops *port_ops; + struct team_port *port; + int ret = 0; + + rcu_read_lock(); + port = team_header_port_get_rcu(team, false); + if (port) { + port_ops = READ_ONCE(port->dev->header_ops); + if (port_ops && port_ops->parse) + ret = port_ops->parse(skb, port->dev, haddr); + } + rcu_read_unlock(); + return ret; +} + +static const struct header_ops team_header_ops = { + .create = team_header_create, + .parse = team_header_parse, +}; + +static void team_setup_by_port(struct net_device *dev, + struct net_device *port_dev) +{ + struct team *team = netdev_priv(dev); + + if (port_dev->type == ARPHRD_ETHER) + dev->header_ops = team->header_ops_cache; + else + dev->header_ops = port_dev->header_ops ? + &team_header_ops : NULL; + dev->type = port_dev->type; + dev->hard_header_len = port_dev->hard_header_len; + dev->needed_headroom = port_dev->needed_headroom; + dev->addr_len = port_dev->addr_len; + dev->mtu = port_dev->mtu; + memcpy(dev->broadcast, port_dev->broadcast, port_dev->addr_len); + eth_hw_addr_inherit(dev, port_dev); + + if (port_dev->flags & IFF_POINTOPOINT) { + dev->flags &= ~(IFF_BROADCAST | IFF_MULTICAST); + dev->flags |= (IFF_POINTOPOINT | IFF_NOARP); + } else if ((port_dev->flags & (IFF_BROADCAST | IFF_MULTICAST)) == + (IFF_BROADCAST | IFF_MULTICAST)) { + dev->flags |= (IFF_BROADCAST | IFF_MULTICAST); + dev->flags &= ~(IFF_POINTOPOINT | IFF_NOARP); + } +} + +static int team_dev_type_check_change(struct net_device *dev, + struct net_device *port_dev) +{ + struct team *team = netdev_priv(dev); + char *portname = port_dev->name; + int err; + + if (dev->type == port_dev->type) + return 0; + if (!list_empty(&team->port_list)) { + netdev_err(dev, "Device %s is of different type\n", portname); + return -EBUSY; + } + err = call_netdevice_notifiers(NETDEV_PRE_TYPE_CHANGE, dev); + err = notifier_to_errno(err); + if (err) { + netdev_err(dev, "Refused to change device type\n"); + return err; + } + dev_uc_flush(dev); + dev_mc_flush(dev); + team_setup_by_port(dev, port_dev); + call_netdevice_notifiers(NETDEV_POST_TYPE_CHANGE, dev); + return 0; +} + +static void team_setup(struct net_device *dev) +{ + struct team *team = netdev_priv(dev); + + ether_setup(dev); + dev->max_mtu = ETH_MAX_MTU; + team->header_ops_cache = dev->header_ops; + + dev->netdev_ops = &team_netdev_ops; + dev->ethtool_ops = &team_ethtool_ops; + dev->needs_free_netdev = true; + dev->priv_destructor = team_destructor; + dev->priv_flags &= ~(IFF_XMIT_DST_RELEASE | IFF_TX_SKB_SHARING); + dev->priv_flags |= IFF_NO_QUEUE; + dev->priv_flags |= IFF_TEAM; + + /* + * Indicate we support unicast address filtering. That way core won't + * bring us to promisc mode in case a unicast addr is added. + * Let this up to underlay drivers. + */ + dev->priv_flags |= IFF_UNICAST_FLT | IFF_LIVE_ADDR_CHANGE; + dev->lltx = true; + + /* Don't allow team devices to change network namespaces. */ + dev->netns_immutable = true; + + dev->features |= NETIF_F_GRO; + + dev->hw_features = MASTER_UPPER_DEV_VLAN_FEATURES | + NETIF_F_HW_VLAN_CTAG_RX | + NETIF_F_HW_VLAN_CTAG_FILTER | + NETIF_F_HW_VLAN_STAG_RX | + NETIF_F_HW_VLAN_STAG_FILTER; + + dev->hw_features |= NETIF_F_GSO_ENCAP_ALL; + dev->features |= dev->hw_features; + dev->features |= NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_STAG_TX; +} + +static int team_newlink(struct net_device *dev, + struct rtnl_newlink_params *params, + struct netlink_ext_ack *extack) +{ + struct nlattr **tb = params->tb; + + if (tb[IFLA_ADDRESS] == NULL) + eth_hw_addr_random(dev); + + return register_netdevice(dev); +} + +static int team_validate(struct nlattr *tb[], struct nlattr *data[], + struct netlink_ext_ack *extack) +{ + if (tb[IFLA_ADDRESS]) { + if (nla_len(tb[IFLA_ADDRESS]) != ETH_ALEN) + return -EINVAL; + if (!is_valid_ether_addr(nla_data(tb[IFLA_ADDRESS]))) + return -EADDRNOTAVAIL; + } + return 0; +} + +static unsigned int team_get_num_tx_queues(void) +{ + return TEAM_DEFAULT_NUM_TX_QUEUES; +} + +static unsigned int team_get_num_rx_queues(void) +{ + return TEAM_DEFAULT_NUM_RX_QUEUES; +} + +static struct rtnl_link_ops team_link_ops __read_mostly = { + .kind = DRV_NAME, + .priv_size = sizeof(struct team), + .setup = team_setup, + .newlink = team_newlink, + .validate = team_validate, + .get_num_tx_queues = team_get_num_tx_queues, + .get_num_rx_queues = team_get_num_rx_queues, +}; + + +/*********************************** + * Generic netlink custom interface + ***********************************/ + +static struct genl_family team_nl_family; + +int team_nl_noop_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct sk_buff *msg; + void *hdr; + int err; + + msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + hdr = genlmsg_put(msg, info->snd_portid, info->snd_seq, + &team_nl_family, 0, TEAM_CMD_NOOP); + if (!hdr) { + err = -EMSGSIZE; + goto err_msg_put; + } + + genlmsg_end(msg, hdr); + + return genlmsg_unicast(genl_info_net(info), msg, info->snd_portid); + +err_msg_put: + nlmsg_free(msg); + + return err; +} + +/* + * Netlink cmd functions should be locked by following two functions. + * Since dev gets held here, that ensures dev won't disappear in between. + */ +static struct team *team_nl_team_get(struct genl_info *info) +{ + struct net *net = genl_info_net(info); + struct net_device *dev; + int ifindex; + + ASSERT_RTNL(); + + if (!info->attrs[TEAM_ATTR_TEAM_IFINDEX]) + return NULL; + + ifindex = nla_get_u32(info->attrs[TEAM_ATTR_TEAM_IFINDEX]); + dev = dev_get_by_index(net, ifindex); + if (!dev || dev->netdev_ops != &team_netdev_ops) { + dev_put(dev); + return NULL; + } + + return netdev_priv(dev); +} + +static void team_nl_team_put(struct team *team) +{ + dev_put(netdev_from_priv(team)); +} + +typedef int team_nl_send_func_t(struct sk_buff *skb, + struct team *team, u32 portid); + +static int team_nl_send_unicast(struct sk_buff *skb, struct team *team, u32 portid) +{ + return genlmsg_unicast(dev_net(netdev_from_priv(team)), skb, portid); +} + +static int team_nl_fill_one_option_get(struct sk_buff *skb, struct team *team, + struct team_option_inst *opt_inst) +{ + struct nlattr *option_item; + struct team_option *option = opt_inst->option; + struct team_option_inst_info *opt_inst_info = &opt_inst->info; + struct team_gsetter_ctx ctx; + int err; + + ctx.info = opt_inst_info; + err = team_option_get(team, opt_inst, &ctx); + if (err) + return err; + + option_item = nla_nest_start_noflag(skb, TEAM_ATTR_ITEM_OPTION); + if (!option_item) + return -EMSGSIZE; + + if (nla_put_string(skb, TEAM_ATTR_OPTION_NAME, option->name)) + goto nest_cancel; + if (opt_inst_info->port && + nla_put_u32(skb, TEAM_ATTR_OPTION_PORT_IFINDEX, + opt_inst_info->port->dev->ifindex)) + goto nest_cancel; + if (opt_inst->option->array_size && + nla_put_u32(skb, TEAM_ATTR_OPTION_ARRAY_INDEX, + opt_inst_info->array_index)) + goto nest_cancel; + + switch (option->type) { + case TEAM_OPTION_TYPE_U32: + if (nla_put_u8(skb, TEAM_ATTR_OPTION_TYPE, NLA_U32)) + goto nest_cancel; + if (nla_put_u32(skb, TEAM_ATTR_OPTION_DATA, ctx.data.u32_val)) + goto nest_cancel; + break; + case TEAM_OPTION_TYPE_STRING: + if (nla_put_u8(skb, TEAM_ATTR_OPTION_TYPE, NLA_STRING)) + goto nest_cancel; + if (nla_put_string(skb, TEAM_ATTR_OPTION_DATA, + ctx.data.str_val)) + goto nest_cancel; + break; + case TEAM_OPTION_TYPE_BINARY: + if (nla_put_u8(skb, TEAM_ATTR_OPTION_TYPE, NLA_BINARY)) + goto nest_cancel; + if (nla_put(skb, TEAM_ATTR_OPTION_DATA, ctx.data.bin_val.len, + ctx.data.bin_val.ptr)) + goto nest_cancel; + break; + case TEAM_OPTION_TYPE_BOOL: + if (nla_put_u8(skb, TEAM_ATTR_OPTION_TYPE, NLA_FLAG)) + goto nest_cancel; + if (ctx.data.bool_val && + nla_put_flag(skb, TEAM_ATTR_OPTION_DATA)) + goto nest_cancel; + break; + case TEAM_OPTION_TYPE_S32: + if (nla_put_u8(skb, TEAM_ATTR_OPTION_TYPE, NLA_S32)) + goto nest_cancel; + if (nla_put_s32(skb, TEAM_ATTR_OPTION_DATA, ctx.data.s32_val)) + goto nest_cancel; + break; + default: + BUG(); + } + if (opt_inst->removed && nla_put_flag(skb, TEAM_ATTR_OPTION_REMOVED)) + goto nest_cancel; + if (opt_inst->changed) { + if (nla_put_flag(skb, TEAM_ATTR_OPTION_CHANGED)) + goto nest_cancel; + opt_inst->changed = false; + } + nla_nest_end(skb, option_item); + return 0; + +nest_cancel: + nla_nest_cancel(skb, option_item); + return -EMSGSIZE; +} + +static int __send_and_alloc_skb(struct sk_buff **pskb, + struct team *team, u32 portid, + team_nl_send_func_t *send_func) +{ + int err; + + if (*pskb) { + err = send_func(*pskb, team, portid); + if (err) + return err; + } + *pskb = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!*pskb) + return -ENOMEM; + return 0; +} + +static int team_nl_send_options_get(struct team *team, u32 portid, u32 seq, + int flags, team_nl_send_func_t *send_func, + struct list_head *sel_opt_inst_list) +{ + struct nlattr *option_list; + struct nlmsghdr *nlh; + void *hdr; + struct team_option_inst *opt_inst; + int err; + struct sk_buff *skb = NULL; + bool incomplete; + int i; + + opt_inst = list_first_entry(sel_opt_inst_list, + struct team_option_inst, tmp_list); + +start_again: + err = __send_and_alloc_skb(&skb, team, portid, send_func); + if (err) + return err; + + hdr = genlmsg_put(skb, portid, seq, &team_nl_family, flags | NLM_F_MULTI, + TEAM_CMD_OPTIONS_GET); + if (!hdr) { + nlmsg_free(skb); + return -EMSGSIZE; + } + + if (nla_put_u32(skb, TEAM_ATTR_TEAM_IFINDEX, + netdev_from_priv(team)->ifindex)) + goto nla_put_failure; + option_list = nla_nest_start_noflag(skb, TEAM_ATTR_LIST_OPTION); + if (!option_list) + goto nla_put_failure; + + i = 0; + incomplete = false; + list_for_each_entry_from(opt_inst, sel_opt_inst_list, tmp_list) { + err = team_nl_fill_one_option_get(skb, team, opt_inst); + if (err) { + if (err == -EMSGSIZE) { + if (!i) + goto errout; + incomplete = true; + break; + } + goto errout; + } + i++; + } + + nla_nest_end(skb, option_list); + genlmsg_end(skb, hdr); + if (incomplete) + goto start_again; + +send_done: + nlh = nlmsg_put(skb, portid, seq, NLMSG_DONE, 0, flags | NLM_F_MULTI); + if (!nlh) { + err = __send_and_alloc_skb(&skb, team, portid, send_func); + if (err) + return err; + goto send_done; + } + + return send_func(skb, team, portid); + +nla_put_failure: + err = -EMSGSIZE; +errout: + nlmsg_free(skb); + return err; +} + +int team_nl_options_get_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct team *team; + struct team_option_inst *opt_inst; + int err; + LIST_HEAD(sel_opt_inst_list); + + rtnl_lock(); + + team = team_nl_team_get(info); + if (!team) { + err = -EINVAL; + goto rtnl_unlock; + } + + list_for_each_entry(opt_inst, &team->option_inst_list, list) + list_add_tail(&opt_inst->tmp_list, &sel_opt_inst_list); + err = team_nl_send_options_get(team, info->snd_portid, info->snd_seq, + NLM_F_ACK, team_nl_send_unicast, + &sel_opt_inst_list); + + team_nl_team_put(team); + +rtnl_unlock: + rtnl_unlock(); + + return err; +} + +static int team_nl_send_event_options_get(struct team *team, + struct list_head *sel_opt_inst_list); + +int team_nl_options_set_doit(struct sk_buff *skb, struct genl_info *info) +{ + struct team *team; + int err = 0; + int i; + struct nlattr *nl_option; + + rtnl_lock(); + + team = team_nl_team_get(info); + if (!team) { + err = -EINVAL; + goto rtnl_unlock; + } + + err = -EINVAL; + if (!info->attrs[TEAM_ATTR_LIST_OPTION]) { + err = -EINVAL; + goto team_put; + } + + nla_for_each_nested(nl_option, info->attrs[TEAM_ATTR_LIST_OPTION], i) { + struct nlattr *opt_attrs[TEAM_ATTR_OPTION_MAX + 1]; + struct nlattr *attr; + struct nlattr *attr_data; + LIST_HEAD(opt_inst_list); + enum team_option_type opt_type; + int opt_port_ifindex = 0; /* != 0 for per-port options */ + u32 opt_array_index = 0; + bool opt_is_array = false; + struct team_option_inst *opt_inst; + char *opt_name; + bool opt_found = false; + + if (nla_type(nl_option) != TEAM_ATTR_ITEM_OPTION) { + err = -EINVAL; + goto team_put; + } + err = nla_parse_nested_deprecated(opt_attrs, + TEAM_ATTR_OPTION_MAX, + nl_option, + team_attr_option_nl_policy, + info->extack); + if (err) + goto team_put; + if (!opt_attrs[TEAM_ATTR_OPTION_NAME] || + !opt_attrs[TEAM_ATTR_OPTION_TYPE]) { + err = -EINVAL; + goto team_put; + } + switch (nla_get_u8(opt_attrs[TEAM_ATTR_OPTION_TYPE])) { + case NLA_U32: + opt_type = TEAM_OPTION_TYPE_U32; + break; + case NLA_STRING: + opt_type = TEAM_OPTION_TYPE_STRING; + break; + case NLA_BINARY: + opt_type = TEAM_OPTION_TYPE_BINARY; + break; + case NLA_FLAG: + opt_type = TEAM_OPTION_TYPE_BOOL; + break; + case NLA_S32: + opt_type = TEAM_OPTION_TYPE_S32; + break; + default: + goto team_put; + } + + attr_data = opt_attrs[TEAM_ATTR_OPTION_DATA]; + if (opt_type != TEAM_OPTION_TYPE_BOOL && !attr_data) { + err = -EINVAL; + goto team_put; + } + + opt_name = nla_data(opt_attrs[TEAM_ATTR_OPTION_NAME]); + attr = opt_attrs[TEAM_ATTR_OPTION_PORT_IFINDEX]; + if (attr) + opt_port_ifindex = nla_get_u32(attr); + + attr = opt_attrs[TEAM_ATTR_OPTION_ARRAY_INDEX]; + if (attr) { + opt_is_array = true; + opt_array_index = nla_get_u32(attr); + } + + list_for_each_entry(opt_inst, &team->option_inst_list, list) { + struct team_option *option = opt_inst->option; + struct team_gsetter_ctx ctx; + struct team_option_inst_info *opt_inst_info; + int tmp_ifindex; + + opt_inst_info = &opt_inst->info; + tmp_ifindex = opt_inst_info->port ? + opt_inst_info->port->dev->ifindex : 0; + if (option->type != opt_type || + strcmp(option->name, opt_name) || + tmp_ifindex != opt_port_ifindex || + (option->array_size && !opt_is_array) || + opt_inst_info->array_index != opt_array_index) + continue; + opt_found = true; + ctx.info = opt_inst_info; + switch (opt_type) { + case TEAM_OPTION_TYPE_U32: + ctx.data.u32_val = nla_get_u32(attr_data); + break; + case TEAM_OPTION_TYPE_STRING: + if (nla_len(attr_data) > TEAM_STRING_MAX_LEN || + !memchr(nla_data(attr_data), '\0', + nla_len(attr_data))) { + err = -EINVAL; + goto team_put; + } + ctx.data.str_val = nla_data(attr_data); + break; + case TEAM_OPTION_TYPE_BINARY: + ctx.data.bin_val.len = nla_len(attr_data); + ctx.data.bin_val.ptr = nla_data(attr_data); + break; + case TEAM_OPTION_TYPE_BOOL: + ctx.data.bool_val = attr_data ? true : false; + break; + case TEAM_OPTION_TYPE_S32: + ctx.data.s32_val = nla_get_s32(attr_data); + break; + default: + BUG(); + } + err = team_option_set(team, opt_inst, &ctx); + if (err) + goto team_put; + opt_inst->changed = true; + list_add(&opt_inst->tmp_list, &opt_inst_list); + } + if (!opt_found) { + err = -ENOENT; + goto team_put; + } + + err = team_nl_send_event_options_get(team, &opt_inst_list); + if (err) + break; + } + +team_put: + team_nl_team_put(team); +rtnl_unlock: + rtnl_unlock(); + return err; +} + +static int team_nl_fill_one_port_get(struct sk_buff *skb, + struct team_port *port) +{ + struct nlattr *port_item; + + port_item = nla_nest_start_noflag(skb, TEAM_ATTR_ITEM_PORT); + if (!port_item) + goto nest_cancel; + if (nla_put_u32(skb, TEAM_ATTR_PORT_IFINDEX, port->dev->ifindex)) + goto nest_cancel; + if (port->changed) { + if (nla_put_flag(skb, TEAM_ATTR_PORT_CHANGED)) + goto nest_cancel; + port->changed = false; + } + if ((port->removed && + nla_put_flag(skb, TEAM_ATTR_PORT_REMOVED)) || + (port->state.linkup && + nla_put_flag(skb, TEAM_ATTR_PORT_LINKUP)) || + nla_put_u32(skb, TEAM_ATTR_PORT_SPEED, port->state.speed) || + nla_put_u8(skb, TEAM_ATTR_PORT_DUPLEX, port->state.duplex)) + goto nest_cancel; + nla_nest_end(skb, port_item); + return 0; + +nest_cancel: + nla_nest_cancel(skb, port_item); + return -EMSGSIZE; +} + +static int team_nl_send_port_list_get(struct team *team, u32 portid, u32 seq, + int flags, team_nl_send_func_t *send_func, + struct team_port *one_port) +{ + struct nlattr *port_list; + struct nlmsghdr *nlh; + void *hdr; + struct team_port *port; + int err; + struct sk_buff *skb = NULL; + bool incomplete; + int i; + + port = list_first_entry_or_null(&team->port_list, + struct team_port, list); + +start_again: + err = __send_and_alloc_skb(&skb, team, portid, send_func); + if (err) + return err; + + hdr = genlmsg_put(skb, portid, seq, &team_nl_family, flags | NLM_F_MULTI, + TEAM_CMD_PORT_LIST_GET); + if (!hdr) { + nlmsg_free(skb); + return -EMSGSIZE; + } + + if (nla_put_u32(skb, TEAM_ATTR_TEAM_IFINDEX, + netdev_from_priv(team)->ifindex)) + goto nla_put_failure; + port_list = nla_nest_start_noflag(skb, TEAM_ATTR_LIST_PORT); + if (!port_list) + goto nla_put_failure; + + i = 0; + incomplete = false; + + /* If one port is selected, called wants to send port list containing + * only this port. Otherwise go through all listed ports and send all + */ + if (one_port) { + err = team_nl_fill_one_port_get(skb, one_port); + if (err) + goto errout; + } else if (port) { + list_for_each_entry_from(port, &team->port_list, list) { + err = team_nl_fill_one_port_get(skb, port); + if (err) { + if (err == -EMSGSIZE) { + if (!i) + goto errout; + incomplete = true; + break; + } + goto errout; + } + i++; + } + } + + nla_nest_end(skb, port_list); + genlmsg_end(skb, hdr); + if (incomplete) + goto start_again; + +send_done: + nlh = nlmsg_put(skb, portid, seq, NLMSG_DONE, 0, flags | NLM_F_MULTI); + if (!nlh) { + err = __send_and_alloc_skb(&skb, team, portid, send_func); + if (err) + return err; + goto send_done; + } + + return send_func(skb, team, portid); + +nla_put_failure: + err = -EMSGSIZE; +errout: + nlmsg_free(skb); + return err; +} + +int team_nl_port_list_get_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct team *team; + int err; + + rtnl_lock(); + + team = team_nl_team_get(info); + if (!team) { + err = -EINVAL; + goto rtnl_unlock; + } + + err = team_nl_send_port_list_get(team, info->snd_portid, info->snd_seq, + NLM_F_ACK, team_nl_send_unicast, NULL); + + team_nl_team_put(team); + +rtnl_unlock: + rtnl_unlock(); + + return err; +} + +static const struct genl_multicast_group team_nl_mcgrps[] = { + { .name = TEAM_GENL_CHANGE_EVENT_MC_GRP_NAME, }, +}; + +static struct genl_family team_nl_family __ro_after_init = { + .name = TEAM_GENL_NAME, + .version = TEAM_GENL_VERSION, + .maxattr = ARRAY_SIZE(team_nl_policy) - 1, + .policy = team_nl_policy, + .netnsok = true, + .module = THIS_MODULE, + .small_ops = team_nl_ops, + .n_small_ops = ARRAY_SIZE(team_nl_ops), + .resv_start_op = TEAM_CMD_PORT_LIST_GET + 1, + .mcgrps = team_nl_mcgrps, + .n_mcgrps = ARRAY_SIZE(team_nl_mcgrps), +}; + +static int team_nl_send_multicast(struct sk_buff *skb, + struct team *team, u32 portid) +{ + return genlmsg_multicast_netns(&team_nl_family, + dev_net(netdev_from_priv(team)), + skb, 0, 0, GFP_KERNEL); +} + +static int team_nl_send_event_options_get(struct team *team, + struct list_head *sel_opt_inst_list) +{ + return team_nl_send_options_get(team, 0, 0, 0, team_nl_send_multicast, + sel_opt_inst_list); +} + +static int team_nl_send_event_port_get(struct team *team, + struct team_port *port) +{ + return team_nl_send_port_list_get(team, 0, 0, 0, team_nl_send_multicast, + port); +} + +static int __init team_nl_init(void) +{ + return genl_register_family(&team_nl_family); +} + +static void __exit team_nl_fini(void) +{ + genl_unregister_family(&team_nl_family); +} + + +/****************** + * Change checkers + ******************/ + +static void __team_options_change_check(struct team *team) +{ + int err; + struct team_option_inst *opt_inst; + LIST_HEAD(sel_opt_inst_list); + + list_for_each_entry(opt_inst, &team->option_inst_list, list) { + if (opt_inst->changed) + list_add_tail(&opt_inst->tmp_list, &sel_opt_inst_list); + } + err = team_nl_send_event_options_get(team, &sel_opt_inst_list); + if (err && err != -ESRCH) + netdev_warn(netdev_from_priv(team), + "Failed to send options change via netlink (err %d)\n", + err); +} + +/* rtnl lock is held */ + +static void __team_port_change_send(struct team_port *port, bool linkup) +{ + int err; + + port->changed = true; + port->state.linkup = linkup; + team_refresh_port_linkup(port); + if (linkup) { + struct ethtool_link_ksettings ecmd; + + err = netif_get_link_ksettings(port->dev, &ecmd); + if (!err) { + port->state.speed = ecmd.base.speed; + port->state.duplex = ecmd.base.duplex; + goto send_event; + } + } + port->state.speed = 0; + port->state.duplex = 0; + +send_event: + err = team_nl_send_event_port_get(port->team, port); + if (err && err != -ESRCH) + netdev_warn(netdev_from_priv(port->team), + "Failed to send port change of device %s via netlink (err %d)\n", + port->dev->name, err); + +} + +static void __team_carrier_check(struct team *team) +{ + struct team_port *port; + bool team_linkup; + + if (team->user_carrier_enabled) + return; + + team_linkup = false; + list_for_each_entry(port, &team->port_list, list) { + if (port->linkup) { + team_linkup = true; + break; + } + } + + if (team_linkup) + netif_carrier_on(netdev_from_priv(team)); + else + netif_carrier_off(netdev_from_priv(team)); +} + +static void __team_port_change_check(struct team_port *port, bool linkup) +{ + if (port->state.linkup != linkup) + __team_port_change_send(port, linkup); + __team_carrier_check(port->team); +} + +static void __team_port_change_port_added(struct team_port *port, bool linkup) +{ + __team_port_change_send(port, linkup); + __team_carrier_check(port->team); +} + +static void __team_port_change_port_removed(struct team_port *port) +{ + port->removed = true; + __team_port_change_send(port, false); + __team_carrier_check(port->team); +} + +static void team_port_change_check(struct team_port *port, bool linkup) +{ + ASSERT_RTNL(); + + __team_port_change_check(port, linkup); +} + + +/************************************ + * Net device notifier event handler + ************************************/ + +static int team_device_event(struct notifier_block *unused, + unsigned long event, void *ptr) +{ + struct net_device *dev = netdev_notifier_info_to_dev(ptr); + struct team_port *port; + + port = team_port_get_rtnl(dev); + if (!port) + return NOTIFY_DONE; + + switch (event) { + case NETDEV_UP: + if (netif_oper_up(dev)) + team_port_change_check(port, true); + break; + case NETDEV_DOWN: + team_port_change_check(port, false); + break; + case NETDEV_CHANGE: + if (netif_running(port->dev)) + team_port_change_check(port, + !!netif_oper_up(port->dev)); + break; + case NETDEV_UNREGISTER: + team_del_slave_on_unregister(netdev_from_priv(port->team), + dev); + break; + case NETDEV_FEAT_CHANGE: + if (!port->team->notifier_ctx) { + port->team->notifier_ctx = true; + netdev_compute_master_upper_features(netdev_from_priv(port->team), + true); + port->team->notifier_ctx = false; + } + break; + case NETDEV_PRECHANGEMTU: + /* Forbid to change mtu of underlaying device */ + if (!port->team->port_mtu_change_allowed) + return NOTIFY_BAD; + break; + case NETDEV_PRE_TYPE_CHANGE: + /* Forbid to change type of underlaying device */ + return NOTIFY_BAD; + case NETDEV_RESEND_IGMP: + /* Propagate to master device */ + call_netdevice_notifiers(event, netdev_from_priv(port->team)); + break; + } + return NOTIFY_DONE; +} + +static struct notifier_block team_notifier_block __read_mostly = { + .notifier_call = team_device_event, +}; + + +/*********************** + * Module init and exit + ***********************/ + +static int __init team_module_init(void) +{ + int err; + + err = register_netdevice_notifier(&team_notifier_block); + if (err) + return err; + + err = rtnl_link_register(&team_link_ops); + if (err) + goto err_rtnl_reg; + + err = team_nl_init(); + if (err) + goto err_nl_init; + + return 0; + +err_nl_init: + rtnl_link_unregister(&team_link_ops); + +err_rtnl_reg: + unregister_netdevice_notifier(&team_notifier_block); + + return err; +} + +static void __exit team_module_exit(void) +{ + team_nl_fini(); + rtnl_link_unregister(&team_link_ops); + unregister_netdevice_notifier(&team_notifier_block); +} + +module_init(team_module_init); +module_exit(team_module_exit); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jiri Pirko "); +MODULE_DESCRIPTION("Ethernet team device driver"); +MODULE_ALIAS_RTNL_LINK(DRV_NAME); +MODULE_IMPORT_NS("NETDEV_INTERNAL"); diff --git a/drivers/net/team/team_mode_activebackup.c b/drivers/net/team/team_mode_activebackup.c new file mode 100644 index 000000000..1c3336c7a --- /dev/null +++ b/drivers/net/team/team_mode_activebackup.c @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * drivers/net/team/team_mode_activebackup.c - Active-backup mode for team + * Copyright (c) 2011 Jiri Pirko + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +struct ab_priv { + struct team_port __rcu *active_port; + struct team_option_inst_info *ap_opt_inst_info; +}; + +static struct ab_priv *ab_priv(struct team *team) +{ + return (struct ab_priv *) &team->mode_priv; +} + +static rx_handler_result_t ab_receive(struct team *team, struct team_port *port, + struct sk_buff *skb) { + struct team_port *active_port; + + active_port = rcu_dereference(ab_priv(team)->active_port); + if (active_port != port) + return RX_HANDLER_EXACT; + return RX_HANDLER_ANOTHER; +} + +static bool ab_transmit(struct team *team, struct sk_buff *skb) +{ + struct team_port *active_port; + + active_port = rcu_dereference_bh(ab_priv(team)->active_port); + if (unlikely(!active_port)) + goto drop; + if (team_dev_queue_xmit(team, active_port, skb)) + return false; + return true; + +drop: + dev_kfree_skb_any(skb); + return false; +} + +static void ab_port_leave(struct team *team, struct team_port *port) +{ + if (ab_priv(team)->active_port == port) { + RCU_INIT_POINTER(ab_priv(team)->active_port, NULL); + team_option_inst_set_change(ab_priv(team)->ap_opt_inst_info); + } +} + +static void ab_active_port_init(struct team *team, + struct team_option_inst_info *info) +{ + ab_priv(team)->ap_opt_inst_info = info; +} + +static void ab_active_port_get(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct team_port *active_port; + + active_port = rtnl_dereference(ab_priv(team)->active_port); + if (active_port) + ctx->data.u32_val = active_port->dev->ifindex; + else + ctx->data.u32_val = 0; +} + +static int ab_active_port_set(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct team_port *port; + + list_for_each_entry(port, &team->port_list, list) { + if (port->dev->ifindex == ctx->data.u32_val) { + rcu_assign_pointer(ab_priv(team)->active_port, port); + return 0; + } + } + return -ENOENT; +} + +static const struct team_option ab_options[] = { + { + .name = "activeport", + .type = TEAM_OPTION_TYPE_U32, + .init = ab_active_port_init, + .getter = ab_active_port_get, + .setter = ab_active_port_set, + }, +}; + +static int ab_init(struct team *team) +{ + return team_options_register(team, ab_options, ARRAY_SIZE(ab_options)); +} + +static void ab_exit(struct team *team) +{ + team_options_unregister(team, ab_options, ARRAY_SIZE(ab_options)); +} + +static const struct team_mode_ops ab_mode_ops = { + .init = ab_init, + .exit = ab_exit, + .receive = ab_receive, + .transmit = ab_transmit, + .port_leave = ab_port_leave, +}; + +static const struct team_mode ab_mode = { + .kind = "activebackup", + .owner = THIS_MODULE, + .priv_size = sizeof(struct ab_priv), + .ops = &ab_mode_ops, + .lag_tx_type = NETDEV_LAG_TX_TYPE_ACTIVEBACKUP, +}; + +static int __init ab_init_module(void) +{ + return team_mode_register(&ab_mode); +} + +static void __exit ab_cleanup_module(void) +{ + team_mode_unregister(&ab_mode); +} + +module_init(ab_init_module); +module_exit(ab_cleanup_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jiri Pirko "); +MODULE_DESCRIPTION("Active-backup mode for team"); +MODULE_ALIAS_TEAM_MODE("activebackup"); diff --git a/drivers/net/team/team_mode_broadcast.c b/drivers/net/team/team_mode_broadcast.c new file mode 100644 index 000000000..61d7d79f0 --- /dev/null +++ b/drivers/net/team/team_mode_broadcast.c @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * drivers/net/team/team_mode_broadcast.c - Broadcast mode for team + * Copyright (c) 2012 Jiri Pirko + */ + +#include +#include +#include +#include +#include +#include + +static bool bc_transmit(struct team *team, struct sk_buff *skb) +{ + struct team_port *cur; + struct team_port *last = NULL; + struct sk_buff *skb2; + bool ret; + bool sum_ret = false; + + list_for_each_entry_rcu(cur, &team->port_list, list) { + if (team_port_txable(cur)) { + if (last) { + skb2 = skb_clone(skb, GFP_ATOMIC); + if (skb2) { + ret = !team_dev_queue_xmit(team, last, + skb2); + if (!sum_ret) + sum_ret = ret; + } + } + last = cur; + } + } + if (last) { + ret = !team_dev_queue_xmit(team, last, skb); + if (!sum_ret) + sum_ret = ret; + } + return sum_ret; +} + +static const struct team_mode_ops bc_mode_ops = { + .transmit = bc_transmit, + .port_enter = team_modeop_port_enter, + .port_change_dev_addr = team_modeop_port_change_dev_addr, +}; + +static const struct team_mode bc_mode = { + .kind = "broadcast", + .owner = THIS_MODULE, + .ops = &bc_mode_ops, + .lag_tx_type = NETDEV_LAG_TX_TYPE_BROADCAST, +}; + +static int __init bc_init_module(void) +{ + return team_mode_register(&bc_mode); +} + +static void __exit bc_cleanup_module(void) +{ + team_mode_unregister(&bc_mode); +} + +module_init(bc_init_module); +module_exit(bc_cleanup_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jiri Pirko "); +MODULE_DESCRIPTION("Broadcast mode for team"); +MODULE_ALIAS_TEAM_MODE("broadcast"); diff --git a/drivers/net/team/team_mode_loadbalance.c b/drivers/net/team/team_mode_loadbalance.c new file mode 100644 index 000000000..38a459649 --- /dev/null +++ b/drivers/net/team/team_mode_loadbalance.c @@ -0,0 +1,698 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * drivers/net/team/team_mode_loadbalance.c - Load-balancing mode for team + * Copyright (c) 2012 Jiri Pirko + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static rx_handler_result_t lb_receive(struct team *team, struct team_port *port, + struct sk_buff *skb) +{ + if (unlikely(skb->protocol == htons(ETH_P_SLOW))) { + /* LACPDU packets should go to exact delivery */ + const unsigned char *dest = eth_hdr(skb)->h_dest; + + if (is_link_local_ether_addr(dest) && dest[5] == 0x02) + return RX_HANDLER_EXACT; + } + return RX_HANDLER_ANOTHER; +} + +struct lb_priv; + +typedef struct team_port *lb_select_tx_port_func_t(struct team *, + unsigned char); + +#define LB_TX_HASHTABLE_SIZE 256 /* hash is a char */ + +struct lb_stats { + u64 tx_bytes; +}; + +struct lb_pcpu_stats { + struct lb_stats hash_stats[LB_TX_HASHTABLE_SIZE]; + struct u64_stats_sync syncp; +}; + +struct lb_stats_info { + struct lb_stats stats; + struct lb_stats last_stats; + struct team_option_inst_info *opt_inst_info; +}; + +struct lb_port_mapping { + struct team_port __rcu *port; + struct team_option_inst_info *opt_inst_info; +}; + +struct lb_priv_ex { + struct team *team; + struct lb_port_mapping tx_hash_to_port_mapping[LB_TX_HASHTABLE_SIZE]; + struct sock_fprog_kern *orig_fprog; + struct { + unsigned int refresh_interval; /* in tenths of second */ + struct delayed_work refresh_dw; + struct lb_stats_info info[LB_TX_HASHTABLE_SIZE]; + } stats; +}; + +struct lb_priv { + struct bpf_prog __rcu *fp; + lb_select_tx_port_func_t __rcu *select_tx_port_func; + struct lb_pcpu_stats __percpu *pcpu_stats; + struct lb_priv_ex *ex; /* priv extension */ +}; + +static struct lb_priv *get_lb_priv(struct team *team) +{ + return (struct lb_priv *) &team->mode_priv; +} + +struct lb_port_priv { + struct lb_stats __percpu *pcpu_stats; + struct lb_stats_info stats_info; +}; + +static struct lb_port_priv *get_lb_port_priv(struct team_port *port) +{ + return (struct lb_port_priv *) &port->mode_priv; +} + +#define LB_HTPM_PORT_BY_HASH(lp_priv, hash) \ + (lb_priv)->ex->tx_hash_to_port_mapping[hash].port + +#define LB_HTPM_OPT_INST_INFO_BY_HASH(lp_priv, hash) \ + (lb_priv)->ex->tx_hash_to_port_mapping[hash].opt_inst_info + +static void lb_tx_hash_to_port_mapping_null_port(struct team *team, + struct team_port *port) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + bool changed = false; + int i; + + for (i = 0; i < LB_TX_HASHTABLE_SIZE; i++) { + struct lb_port_mapping *pm; + + pm = &lb_priv->ex->tx_hash_to_port_mapping[i]; + if (rcu_access_pointer(pm->port) == port) { + RCU_INIT_POINTER(pm->port, NULL); + team_option_inst_set_change(pm->opt_inst_info); + changed = true; + } + } + if (changed) + team_options_change_check(team); +} + +/* Basic tx selection based solely by hash */ +static struct team_port *lb_hash_select_tx_port(struct team *team, + unsigned char hash) +{ + int port_index = team_num_to_port_index(team, hash); + + return team_get_port_by_tx_index_rcu(team, port_index); +} + +/* Hash to port mapping select tx port */ +static struct team_port *lb_htpm_select_tx_port(struct team *team, + unsigned char hash) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + struct team_port *port; + + port = rcu_dereference_bh(LB_HTPM_PORT_BY_HASH(lb_priv, hash)); + if (likely(port)) + return port; + /* If no valid port in the table, fall back to simple hash */ + return lb_hash_select_tx_port(team, hash); +} + +struct lb_select_tx_port { + char *name; + lb_select_tx_port_func_t *func; +}; + +static const struct lb_select_tx_port lb_select_tx_port_list[] = { + { + .name = "hash", + .func = lb_hash_select_tx_port, + }, + { + .name = "hash_to_port_mapping", + .func = lb_htpm_select_tx_port, + }, +}; +#define LB_SELECT_TX_PORT_LIST_COUNT ARRAY_SIZE(lb_select_tx_port_list) + +static char *lb_select_tx_port_get_name(lb_select_tx_port_func_t *func) +{ + int i; + + for (i = 0; i < LB_SELECT_TX_PORT_LIST_COUNT; i++) { + const struct lb_select_tx_port *item; + + item = &lb_select_tx_port_list[i]; + if (item->func == func) + return item->name; + } + return NULL; +} + +static lb_select_tx_port_func_t *lb_select_tx_port_get_func(const char *name) +{ + int i; + + for (i = 0; i < LB_SELECT_TX_PORT_LIST_COUNT; i++) { + const struct lb_select_tx_port *item; + + item = &lb_select_tx_port_list[i]; + if (!strcmp(item->name, name)) + return item->func; + } + return NULL; +} + +static unsigned int lb_get_skb_hash(struct lb_priv *lb_priv, + struct sk_buff *skb) +{ + struct bpf_prog *fp; + uint32_t lhash; + unsigned char *c; + + fp = rcu_dereference_bh(lb_priv->fp); + if (unlikely(!fp)) + return 0; + lhash = bpf_prog_run(fp, skb); + c = (char *) &lhash; + return c[0] ^ c[1] ^ c[2] ^ c[3]; +} + +static void lb_update_tx_stats(unsigned int tx_bytes, struct lb_priv *lb_priv, + struct lb_port_priv *lb_port_priv, + unsigned char hash) +{ + struct lb_pcpu_stats *pcpu_stats; + struct lb_stats *port_stats; + struct lb_stats *hash_stats; + + pcpu_stats = this_cpu_ptr(lb_priv->pcpu_stats); + port_stats = this_cpu_ptr(lb_port_priv->pcpu_stats); + hash_stats = &pcpu_stats->hash_stats[hash]; + u64_stats_update_begin(&pcpu_stats->syncp); + port_stats->tx_bytes += tx_bytes; + hash_stats->tx_bytes += tx_bytes; + u64_stats_update_end(&pcpu_stats->syncp); +} + +static bool lb_transmit(struct team *team, struct sk_buff *skb) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + lb_select_tx_port_func_t *select_tx_port_func; + struct team_port *port; + unsigned char hash; + unsigned int tx_bytes = skb->len; + + hash = lb_get_skb_hash(lb_priv, skb); + select_tx_port_func = rcu_dereference_bh(lb_priv->select_tx_port_func); + port = select_tx_port_func(team, hash); + if (unlikely(!port)) + goto drop; + if (team_dev_queue_xmit(team, port, skb)) + return false; + lb_update_tx_stats(tx_bytes, lb_priv, get_lb_port_priv(port), hash); + return true; + +drop: + dev_kfree_skb_any(skb); + return false; +} + +static void lb_bpf_func_get(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + + if (!lb_priv->ex->orig_fprog) { + ctx->data.bin_val.len = 0; + ctx->data.bin_val.ptr = NULL; + return; + } + ctx->data.bin_val.len = lb_priv->ex->orig_fprog->len * + sizeof(struct sock_filter); + ctx->data.bin_val.ptr = lb_priv->ex->orig_fprog->filter; +} + +static int __fprog_create(struct sock_fprog_kern **pfprog, u32 data_len, + const void *data) +{ + struct sock_fprog_kern *fprog; + struct sock_filter *filter = (struct sock_filter *) data; + + if (data_len % sizeof(struct sock_filter)) + return -EINVAL; + fprog = kmalloc_obj(*fprog); + if (!fprog) + return -ENOMEM; + fprog->filter = kmemdup(filter, data_len, GFP_KERNEL); + if (!fprog->filter) { + kfree(fprog); + return -ENOMEM; + } + fprog->len = data_len / sizeof(struct sock_filter); + *pfprog = fprog; + return 0; +} + +static void __fprog_destroy(struct sock_fprog_kern *fprog) +{ + kfree(fprog->filter); + kfree(fprog); +} + +static int lb_bpf_func_set(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + struct bpf_prog *fp = NULL; + struct bpf_prog *orig_fp = NULL; + struct sock_fprog_kern *fprog = NULL; + int err; + + if (ctx->data.bin_val.len) { + err = __fprog_create(&fprog, ctx->data.bin_val.len, + ctx->data.bin_val.ptr); + if (err) + return err; + err = bpf_prog_create(&fp, fprog); + if (err) { + __fprog_destroy(fprog); + return err; + } + } + + if (lb_priv->ex->orig_fprog) { + /* Clear old filter data */ + __fprog_destroy(lb_priv->ex->orig_fprog); + orig_fp = rtnl_dereference(lb_priv->fp); + } + + rcu_assign_pointer(lb_priv->fp, fp); + lb_priv->ex->orig_fprog = fprog; + + if (orig_fp) { + synchronize_rcu(); + bpf_prog_destroy(orig_fp); + } + return 0; +} + +static void lb_bpf_func_free(struct team *team) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + struct bpf_prog *fp; + + if (!lb_priv->ex->orig_fprog) + return; + + __fprog_destroy(lb_priv->ex->orig_fprog); + fp = rtnl_dereference(lb_priv->fp); + bpf_prog_destroy(fp); +} + +static void lb_tx_method_get(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + lb_select_tx_port_func_t *func; + char *name; + + func = rtnl_dereference(lb_priv->select_tx_port_func); + name = lb_select_tx_port_get_name(func); + BUG_ON(!name); + ctx->data.str_val = name; +} + +static int lb_tx_method_set(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + lb_select_tx_port_func_t *func; + + func = lb_select_tx_port_get_func(ctx->data.str_val); + if (!func) + return -EINVAL; + rcu_assign_pointer(lb_priv->select_tx_port_func, func); + return 0; +} + +static void lb_tx_hash_to_port_mapping_init(struct team *team, + struct team_option_inst_info *info) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + unsigned char hash = info->array_index; + + LB_HTPM_OPT_INST_INFO_BY_HASH(lb_priv, hash) = info; +} + +static void lb_tx_hash_to_port_mapping_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + struct team_port *port; + unsigned char hash = ctx->info->array_index; + + port = LB_HTPM_PORT_BY_HASH(lb_priv, hash); + ctx->data.u32_val = port ? port->dev->ifindex : 0; +} + +static int lb_tx_hash_to_port_mapping_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + struct team_port *port; + unsigned char hash = ctx->info->array_index; + + list_for_each_entry(port, &team->port_list, list) { + if (ctx->data.u32_val == port->dev->ifindex && + team_port_tx_enabled(port)) { + rcu_assign_pointer(LB_HTPM_PORT_BY_HASH(lb_priv, hash), + port); + return 0; + } + } + return -ENODEV; +} + +static void lb_hash_stats_init(struct team *team, + struct team_option_inst_info *info) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + unsigned char hash = info->array_index; + + lb_priv->ex->stats.info[hash].opt_inst_info = info; +} + +static void lb_hash_stats_get(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + unsigned char hash = ctx->info->array_index; + + ctx->data.bin_val.ptr = &lb_priv->ex->stats.info[hash].stats; + ctx->data.bin_val.len = sizeof(struct lb_stats); +} + +static void lb_port_stats_init(struct team *team, + struct team_option_inst_info *info) +{ + struct team_port *port = info->port; + struct lb_port_priv *lb_port_priv = get_lb_port_priv(port); + + lb_port_priv->stats_info.opt_inst_info = info; +} + +static void lb_port_stats_get(struct team *team, struct team_gsetter_ctx *ctx) +{ + struct team_port *port = ctx->info->port; + struct lb_port_priv *lb_port_priv = get_lb_port_priv(port); + + ctx->data.bin_val.ptr = &lb_port_priv->stats_info.stats; + ctx->data.bin_val.len = sizeof(struct lb_stats); +} + +static void __lb_stats_info_refresh_prepare(struct lb_stats_info *s_info) +{ + memcpy(&s_info->last_stats, &s_info->stats, sizeof(struct lb_stats)); + memset(&s_info->stats, 0, sizeof(struct lb_stats)); +} + +static bool __lb_stats_info_refresh_check(struct lb_stats_info *s_info, + struct team *team) +{ + if (memcmp(&s_info->last_stats, &s_info->stats, + sizeof(struct lb_stats))) { + team_option_inst_set_change(s_info->opt_inst_info); + return true; + } + return false; +} + +static void __lb_one_cpu_stats_add(struct lb_stats *acc_stats, + struct lb_stats *cpu_stats, + struct u64_stats_sync *syncp) +{ + unsigned int start; + struct lb_stats tmp; + + do { + start = u64_stats_fetch_begin(syncp); + tmp.tx_bytes = cpu_stats->tx_bytes; + } while (u64_stats_fetch_retry(syncp, start)); + acc_stats->tx_bytes += tmp.tx_bytes; +} + +static void lb_stats_refresh(struct work_struct *work) +{ + struct team *team; + struct lb_priv *lb_priv; + struct lb_priv_ex *lb_priv_ex; + struct lb_pcpu_stats *pcpu_stats; + struct lb_stats *stats; + struct lb_stats_info *s_info; + struct team_port *port; + bool changed = false; + int i; + int j; + + lb_priv_ex = container_of(work, struct lb_priv_ex, + stats.refresh_dw.work); + + team = lb_priv_ex->team; + lb_priv = get_lb_priv(team); + + if (!rtnl_trylock()) { + schedule_delayed_work(&lb_priv_ex->stats.refresh_dw, 0); + return; + } + + for (j = 0; j < LB_TX_HASHTABLE_SIZE; j++) { + s_info = &lb_priv->ex->stats.info[j]; + __lb_stats_info_refresh_prepare(s_info); + for_each_possible_cpu(i) { + pcpu_stats = per_cpu_ptr(lb_priv->pcpu_stats, i); + stats = &pcpu_stats->hash_stats[j]; + __lb_one_cpu_stats_add(&s_info->stats, stats, + &pcpu_stats->syncp); + } + changed |= __lb_stats_info_refresh_check(s_info, team); + } + + list_for_each_entry(port, &team->port_list, list) { + struct lb_port_priv *lb_port_priv = get_lb_port_priv(port); + + s_info = &lb_port_priv->stats_info; + __lb_stats_info_refresh_prepare(s_info); + for_each_possible_cpu(i) { + pcpu_stats = per_cpu_ptr(lb_priv->pcpu_stats, i); + stats = per_cpu_ptr(lb_port_priv->pcpu_stats, i); + __lb_one_cpu_stats_add(&s_info->stats, stats, + &pcpu_stats->syncp); + } + changed |= __lb_stats_info_refresh_check(s_info, team); + } + + if (changed) + team_options_change_check(team); + + schedule_delayed_work(&lb_priv_ex->stats.refresh_dw, + (lb_priv_ex->stats.refresh_interval * HZ) / 10); + + rtnl_unlock(); +} + +static void lb_stats_refresh_interval_get(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + + ctx->data.u32_val = lb_priv->ex->stats.refresh_interval; +} + +static int lb_stats_refresh_interval_set(struct team *team, + struct team_gsetter_ctx *ctx) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + unsigned int interval; + + interval = ctx->data.u32_val; + if (lb_priv->ex->stats.refresh_interval == interval) + return 0; + lb_priv->ex->stats.refresh_interval = interval; + if (interval) + schedule_delayed_work(&lb_priv->ex->stats.refresh_dw, 0); + else + cancel_delayed_work(&lb_priv->ex->stats.refresh_dw); + return 0; +} + +static const struct team_option lb_options[] = { + { + .name = "bpf_hash_func", + .type = TEAM_OPTION_TYPE_BINARY, + .getter = lb_bpf_func_get, + .setter = lb_bpf_func_set, + }, + { + .name = "lb_tx_method", + .type = TEAM_OPTION_TYPE_STRING, + .getter = lb_tx_method_get, + .setter = lb_tx_method_set, + }, + { + .name = "lb_tx_hash_to_port_mapping", + .array_size = LB_TX_HASHTABLE_SIZE, + .type = TEAM_OPTION_TYPE_U32, + .init = lb_tx_hash_to_port_mapping_init, + .getter = lb_tx_hash_to_port_mapping_get, + .setter = lb_tx_hash_to_port_mapping_set, + }, + { + .name = "lb_hash_stats", + .array_size = LB_TX_HASHTABLE_SIZE, + .type = TEAM_OPTION_TYPE_BINARY, + .init = lb_hash_stats_init, + .getter = lb_hash_stats_get, + }, + { + .name = "lb_port_stats", + .per_port = true, + .type = TEAM_OPTION_TYPE_BINARY, + .init = lb_port_stats_init, + .getter = lb_port_stats_get, + }, + { + .name = "lb_stats_refresh_interval", + .type = TEAM_OPTION_TYPE_U32, + .getter = lb_stats_refresh_interval_get, + .setter = lb_stats_refresh_interval_set, + }, +}; + +static int lb_init(struct team *team) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + lb_select_tx_port_func_t *func; + int i, err; + + /* set default tx port selector */ + func = lb_select_tx_port_get_func("hash"); + BUG_ON(!func); + rcu_assign_pointer(lb_priv->select_tx_port_func, func); + + lb_priv->ex = kzalloc_obj(*lb_priv->ex); + if (!lb_priv->ex) + return -ENOMEM; + lb_priv->ex->team = team; + + lb_priv->pcpu_stats = alloc_percpu(struct lb_pcpu_stats); + if (!lb_priv->pcpu_stats) { + err = -ENOMEM; + goto err_alloc_pcpu_stats; + } + + for_each_possible_cpu(i) { + struct lb_pcpu_stats *team_lb_stats; + team_lb_stats = per_cpu_ptr(lb_priv->pcpu_stats, i); + u64_stats_init(&team_lb_stats->syncp); + } + + + INIT_DELAYED_WORK(&lb_priv->ex->stats.refresh_dw, lb_stats_refresh); + + err = team_options_register(team, lb_options, ARRAY_SIZE(lb_options)); + if (err) + goto err_options_register; + return 0; + +err_options_register: + free_percpu(lb_priv->pcpu_stats); +err_alloc_pcpu_stats: + kfree(lb_priv->ex); + return err; +} + +static void lb_exit(struct team *team) +{ + struct lb_priv *lb_priv = get_lb_priv(team); + + team_options_unregister(team, lb_options, + ARRAY_SIZE(lb_options)); + lb_bpf_func_free(team); + cancel_delayed_work_sync(&lb_priv->ex->stats.refresh_dw); + free_percpu(lb_priv->pcpu_stats); + kfree(lb_priv->ex); +} + +static int lb_port_enter(struct team *team, struct team_port *port) +{ + struct lb_port_priv *lb_port_priv = get_lb_port_priv(port); + + lb_port_priv->pcpu_stats = alloc_percpu(struct lb_stats); + if (!lb_port_priv->pcpu_stats) + return -ENOMEM; + return 0; +} + +static void lb_port_leave(struct team *team, struct team_port *port) +{ + struct lb_port_priv *lb_port_priv = get_lb_port_priv(port); + + free_percpu(lb_port_priv->pcpu_stats); +} + +static void lb_port_tx_disabled(struct team *team, struct team_port *port) +{ + lb_tx_hash_to_port_mapping_null_port(team, port); +} + +static const struct team_mode_ops lb_mode_ops = { + .init = lb_init, + .exit = lb_exit, + .port_enter = lb_port_enter, + .port_leave = lb_port_leave, + .port_tx_disabled = lb_port_tx_disabled, + .receive = lb_receive, + .transmit = lb_transmit, +}; + +static const struct team_mode lb_mode = { + .kind = "loadbalance", + .owner = THIS_MODULE, + .priv_size = sizeof(struct lb_priv), + .port_priv_size = sizeof(struct lb_port_priv), + .ops = &lb_mode_ops, + .lag_tx_type = NETDEV_LAG_TX_TYPE_HASH, +}; + +static int __init lb_init_module(void) +{ + return team_mode_register(&lb_mode); +} + +static void __exit lb_cleanup_module(void) +{ + team_mode_unregister(&lb_mode); +} + +module_init(lb_init_module); +module_exit(lb_cleanup_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jiri Pirko "); +MODULE_DESCRIPTION("Load-balancing mode for team"); +MODULE_ALIAS_TEAM_MODE("loadbalance"); diff --git a/drivers/net/team/team_mode_random.c b/drivers/net/team/team_mode_random.c new file mode 100644 index 000000000..370e974f3 --- /dev/null +++ b/drivers/net/team/team_mode_random.c @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * drivers/net/team/team_mode_random.c - Random mode for team + * Copyright (c) 2013 Jiri Pirko + */ + +#include +#include +#include +#include +#include +#include + +static bool rnd_transmit(struct team *team, struct sk_buff *skb) +{ + struct team_port *port; + int port_index; + + port_index = get_random_u32_below(READ_ONCE(team->tx_en_port_count)); + port = team_get_port_by_tx_index_rcu(team, port_index); + if (unlikely(!port)) + goto drop; + port = team_get_first_port_txable_rcu(team, port); + if (unlikely(!port)) + goto drop; + if (team_dev_queue_xmit(team, port, skb)) + return false; + return true; + +drop: + dev_kfree_skb_any(skb); + return false; +} + +static const struct team_mode_ops rnd_mode_ops = { + .transmit = rnd_transmit, + .port_enter = team_modeop_port_enter, + .port_change_dev_addr = team_modeop_port_change_dev_addr, +}; + +static const struct team_mode rnd_mode = { + .kind = "random", + .owner = THIS_MODULE, + .ops = &rnd_mode_ops, + .lag_tx_type = NETDEV_LAG_TX_TYPE_RANDOM, +}; + +static int __init rnd_init_module(void) +{ + return team_mode_register(&rnd_mode); +} + +static void __exit rnd_cleanup_module(void) +{ + team_mode_unregister(&rnd_mode); +} + +module_init(rnd_init_module); +module_exit(rnd_cleanup_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jiri Pirko "); +MODULE_DESCRIPTION("Random mode for team"); +MODULE_ALIAS_TEAM_MODE("random"); diff --git a/drivers/net/team/team_mode_roundrobin.c b/drivers/net/team/team_mode_roundrobin.c new file mode 100644 index 000000000..ecbeef28c --- /dev/null +++ b/drivers/net/team/team_mode_roundrobin.c @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * drivers/net/team/team_mode_roundrobin.c - Round-robin mode for team + * Copyright (c) 2011 Jiri Pirko + */ + +#include +#include +#include +#include +#include +#include + +struct rr_priv { + unsigned int sent_packets; +}; + +static struct rr_priv *rr_priv(struct team *team) +{ + return (struct rr_priv *) &team->mode_priv; +} + +static bool rr_transmit(struct team *team, struct sk_buff *skb) +{ + struct team_port *port; + int port_index; + + port_index = team_num_to_port_index(team, + rr_priv(team)->sent_packets++); + port = team_get_port_by_tx_index_rcu(team, port_index); + if (unlikely(!port)) + goto drop; + port = team_get_first_port_txable_rcu(team, port); + if (unlikely(!port)) + goto drop; + if (team_dev_queue_xmit(team, port, skb)) + return false; + return true; + +drop: + dev_kfree_skb_any(skb); + return false; +} + +static const struct team_mode_ops rr_mode_ops = { + .transmit = rr_transmit, + .port_enter = team_modeop_port_enter, + .port_change_dev_addr = team_modeop_port_change_dev_addr, +}; + +static const struct team_mode rr_mode = { + .kind = "roundrobin", + .owner = THIS_MODULE, + .priv_size = sizeof(struct rr_priv), + .ops = &rr_mode_ops, + .lag_tx_type = NETDEV_LAG_TX_TYPE_ROUNDROBIN, +}; + +static int __init rr_init_module(void) +{ + return team_mode_register(&rr_mode); +} + +static void __exit rr_cleanup_module(void) +{ + team_mode_unregister(&rr_mode); +} + +module_init(rr_init_module); +module_exit(rr_cleanup_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jiri Pirko "); +MODULE_DESCRIPTION("Round-robin mode for team"); +MODULE_ALIAS_TEAM_MODE("roundrobin"); diff --git a/drivers/net/team/team_nl.c b/drivers/net/team/team_nl.c new file mode 100644 index 000000000..6db21725f --- /dev/null +++ b/drivers/net/team/team_nl.c @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/team.yaml */ +/* YNL-GEN kernel source */ +/* To regenerate run: tools/net/ynl/ynl-regen.sh */ + +#include +#include + +#include "team_nl.h" + +#include + +/* Common nested types */ +const struct nla_policy team_attr_option_nl_policy[TEAM_ATTR_OPTION_ARRAY_INDEX + 1] = { + [TEAM_ATTR_OPTION_NAME] = { .type = NLA_STRING, .len = TEAM_STRING_MAX_LEN, }, + [TEAM_ATTR_OPTION_CHANGED] = { .type = NLA_FLAG, }, + [TEAM_ATTR_OPTION_TYPE] = { .type = NLA_U8, }, + [TEAM_ATTR_OPTION_DATA] = { .type = NLA_BINARY, }, + [TEAM_ATTR_OPTION_REMOVED] = { .type = NLA_FLAG, }, + [TEAM_ATTR_OPTION_PORT_IFINDEX] = { .type = NLA_U32, }, + [TEAM_ATTR_OPTION_ARRAY_INDEX] = { .type = NLA_U32, }, +}; + +const struct nla_policy team_item_option_nl_policy[TEAM_ATTR_ITEM_OPTION + 1] = { + [TEAM_ATTR_ITEM_OPTION] = NLA_POLICY_NESTED(team_attr_option_nl_policy), +}; + +/* Global operation policy for team */ +const struct nla_policy team_nl_policy[TEAM_ATTR_LIST_OPTION + 1] = { + [TEAM_ATTR_TEAM_IFINDEX] = { .type = NLA_U32, }, + [TEAM_ATTR_LIST_OPTION] = NLA_POLICY_NESTED(team_item_option_nl_policy), +}; + +/* Ops table for team */ +const struct genl_small_ops team_nl_ops[4] = { + { + .cmd = TEAM_CMD_NOOP, + .validate = GENL_DONT_VALIDATE_STRICT, + .doit = team_nl_noop_doit, + }, + { + .cmd = TEAM_CMD_OPTIONS_SET, + .validate = GENL_DONT_VALIDATE_STRICT, + .doit = team_nl_options_set_doit, + .flags = GENL_ADMIN_PERM, + }, + { + .cmd = TEAM_CMD_OPTIONS_GET, + .validate = GENL_DONT_VALIDATE_STRICT, + .doit = team_nl_options_get_doit, + .flags = GENL_ADMIN_PERM, + }, + { + .cmd = TEAM_CMD_PORT_LIST_GET, + .validate = GENL_DONT_VALIDATE_STRICT, + .doit = team_nl_port_list_get_doit, + .flags = GENL_ADMIN_PERM, + }, +}; diff --git a/drivers/net/team/team_nl.h b/drivers/net/team/team_nl.h new file mode 100644 index 000000000..74816b193 --- /dev/null +++ b/drivers/net/team/team_nl.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/team.yaml */ +/* YNL-GEN kernel header */ +/* To regenerate run: tools/net/ynl/ynl-regen.sh */ + +#ifndef _LINUX_TEAM_GEN_H +#define _LINUX_TEAM_GEN_H + +#include +#include + +#include + +/* Common nested types */ +extern const struct nla_policy team_attr_option_nl_policy[TEAM_ATTR_OPTION_ARRAY_INDEX + 1]; +extern const struct nla_policy team_item_option_nl_policy[TEAM_ATTR_ITEM_OPTION + 1]; + +/* Global operation policy for team */ +extern const struct nla_policy team_nl_policy[TEAM_ATTR_LIST_OPTION + 1]; + +/* Ops table for team */ +extern const struct genl_small_ops team_nl_ops[4]; + +int team_nl_noop_doit(struct sk_buff *skb, struct genl_info *info); +int team_nl_options_set_doit(struct sk_buff *skb, struct genl_info *info); +int team_nl_options_get_doit(struct sk_buff *skb, struct genl_info *info); +int team_nl_port_list_get_doit(struct sk_buff *skb, struct genl_info *info); + +#endif /* _LINUX_TEAM_GEN_H */ -- cgit v1.3.1