IABSD.fr/src/sys/netinet
Branch :
Log
| Author | Commit | Date | CI | Message |
|---|---|---|---|---|
 |
4039bfa0 | 2025-02-17 20:31:25 | Handle RTF_GATEWAY route with rt_gwroute NULL. rtrequest_delete() calls rt_putgwroute() to set rt_gwroute to NULL. When another thread holds a reference to such a route, an assertion failed in rtisvalid() and rt_getll(). Handle this case, rt_getll() may return NULL then. OK claudio@ | |
|
7a23ec91 | 2025-02-17 12:46:02 | Toeplitz hash for UDP and IPv6 TCP output. IPv4 TCP output uses the toeplitz hash as flow id. It is calculated in in_pcbconnect() and can be used for all connected sockets. Add it for UDP and TCP IPv6, too. As pf calculates its own hash, this affects only setups with pf disabled. It gives an improvement in traffic distribution over the queues and 20% performance increase with UDP send on v4/v6 and TCP send on v6 without pf. tested and OK sf@ | |
 |
2f5a0aea | 2025-02-17 08:56:33 | Get rid of unused `so' argument in sbreserve(). ok bluhm | |
|
3411e7e2 | 2025-02-16 11:39:28 | Revert SMR protection of rt_gwroute. Using a smr_barrier() in rt_putgwroute() slows down adding routes. This is the hot path for BGP router. Syncing the FIB is now taking ages and the system is close to unrespnsive in that time. found by claudio@ | |
 |
51442b8a | 2025-02-14 13:14:13 | add tunneldf support to sec(4) sec(4) is a very thin wrapper around the existing ipsec output processing for encapsulating packets, and inherited the behaviour that the DF flag was propagated from the encapsulated packet to the outer ip header. this means if the sec(4) interface has a large mtu and is carrying packets with DF set over a network that can't transport large(r) packets, these packets are effectively dropped. ipsec applied via the SPD copes with this by having SAs figure out the path mtu and using that when applying policy, but sec(4) is an interface, so the network stack uses the interface mtu rather than the associated SA path mtu. rfc4459 discusses this kind of problem has offers a variety of solutions. this implements one of the simpler options, which is to allow the tunnel endpoints to manage the DF regardless of the payload and reassemble the encapsulated packets. to actually do this, ipsec output packet processing has to be able to take an argument that says how you want DF to be handled. in the future we're going to look at how we can use the path mtu determined by the ipsec SA to try and implement one of the other solutions from the RFC, which is to signal the lower mtu to the sources of tunnelled packets. tested by and ok claudio@ | |
|
dddbedba | 2025-02-13 21:01:34 | Fix route entry race when accessing rt_gwroute. Kassert in rt_getll() was triggered as rt_gwroute could be NULL. Problem was introduced by shared netlock around tcp_timer_rexmt(). PMTU discovery calls rtrequest_delete() which was missing proper locking around rt_gwroute. As rt_getll() is called by ARP and ND6 resolve in the hot path, use SMR to provide the pointer to rt_gwroute lockless. Reference count of the returned route is incremented, caller has to free it. Modifying rt_gwroute or rt_cachecnt in rt_putgwroute() is protected by per route lock. OK mvs@ | |
|
bd92615b | 2025-02-12 21:28:10 | Use socket lock for inpcb notify. The notify and ctlinput functions were not MP safe. They need socket lock which can be aquired by in_pcbsolock_ref(). Of course in_pcbnotifyall() has to be called without holding a socket lock. Rename in_rtchange() to in_pcbrtchange(). This is the correct namespace and the functions take care of the inpcb route. OK mvs@ | |
|
ba453f40 | 2025-02-10 15:06:57 | Fix TCP maximum segment size with IPsec. When IPsec is used, if_get(m->m_pkthdr.ph_ifidx) returns enc0. Its if_mtu is 0 which results in negative mss. After fixing a signed integer comparison bug with imax(), tcp_mss_adv() used mssdflt, which is 512. So the TCP SYN cache sent packets with a small maximum TCP segment number. The underlying problem is, that SYN cache used the incoming interface m->m_pkthdr.ph_ifidx for the outgoing MTU. The correct way is to use the route of the destination address like tcp_mss() does it. The SYN cache has a struct route which can be used. An additional route lookup does not happen as the route is cached and will be reused by ip_output(). OK mvs@ | |
|
1a5460a4 | 2025-02-06 23:53:55 | Never install path MTU routes for IPsec transport mode SAs. Prevent installation of PMTU-routes for transport mode ESP-SAs in both cases when ip_output_ipsec_pmtu_update() gets called. from markus@ | |
|
ea68d214 | 2025-02-06 13:40:57 | Call pru_attach() with shared solock() within socreate(). The internal internet attach functions look MP safe. Do external unlocking and release within error path because sofree() relies on exclusive solock(). ok bluhm | |
|
e489f29b | 2025-02-06 13:39:31 | Get rid of unused `so' argument in sbflush(). No functional changes. ok bluhm | |
|
8c2fa8a9 | 2025-02-05 10:15:10 | Fix race in inpcb mutex to socket lock conversion. Testing parallel TCP input revealed a race in in_pcbsolock_ref(). The mutex inp_sofree_mtx is used to reliably get the socket from the incpb and refcount it. Then the socket lock is used to prevent further calls to in_pcbdetach() or sofree(). But between releasing the inpcb mutex and acquiring the socket lock, the inpcb could detach. So when holding the socket lock reassure that the inpcb is still associated by the socket. Otherwise locking the socket belonging to the inpcb has failed. OK mvs@ | |
|
5d1d4693 | 2025-01-31 11:48:18 | Get rid of unused `so' argument in sbdrop(). No functional changes. ok bluhm | |
|
1d90c3fb | 2025-01-30 14:40:50 | Get rid of unused `so' argument in sbspace(). No functional changes. ok bluhm | |
|
fd7b854c | 2025-01-30 08:52:33 | Get rid of unused `so' argument in sb_notify(). No functional changes. ok bluhm | |
|
5d680383 | 2025-01-26 17:21:26 | Syn cache calls TCP drop instead of socket abort. Instead of calling socket layer soabort() and then down via tcp_abort() which ends in tcp_drop(), call tcp_drop() directly from the TCP syn cache. The errno is not relevant as the new socket is dropped before it can be reached from userland. OK mvs@ | |
|
6c6f11bf | 2025-01-25 23:55:32 | Rename old socket to more specific listen socket in TCP syn cache. OK mvs@ | |
|
e2bf3321 | 2025-01-25 22:06:41 | Keep socket lock in sonewconn() for new connection. For TCP input unlocking we need a consistent lock of the newly created socket. Instead of releasing the lock in sonewconn() and grabbing it again later, it is better that sonewconn() returns a locked socket. For now only change syn_cache_get() which calls in_pcbsounlock_rele() at the end. Following diffs will push the unlock into tcp_input(). OK mvs@ | |
 |
f168c03c | 2025-01-25 02:06:40 | Check the source address for the tunneled packets. ok mvs | |
|
92546b59 | 2025-01-23 12:51:51 | Fix inpcb leak in divert attach. All other internet socket attach functions first call soreserve() and then in_pcballoc(). This avoids an in_pcbdetach() in the error path. Current divert attach code may leak the inpcb. Reorder calls to allow simple error handling. OK mvs@ | |
|
cbb583bb | 2025-01-22 09:37:06 | Convert bcopy() to memcpy() in tcp_respond(). Struct ip, ip6, and th point to locations on m, which is new memory from m_gethdr(). There is no overlapping memory, so use memcpy. from dhill@; OK mvs@ | |
|
e835bce2 | 2025-01-16 11:59:20 | Remove net lock from TCP sysctl for keep alive. Keep copies in seconds for the sysctl and update timer variables atomically when they change. tcp_maxidle was historically calculated in tcp_slowtimo() as the timers were called from there. Better calculate maxidle when needed. tcp_timer_init() is useless, just initialize data. While there make the names consistent. input sthen@; OK mvs@ | |
|
680a5d21 | 2025-01-14 13:49:44 | Remove exclusive net lock from TCP timers. TCP timers can run with shared netlock and socket lock. Use common tcp_timer_enter() and tcp_timer_leave() that lock the socket and do reference counting. Then incpb and socket always exist. input and OK mvs@ | |
|
5d0c3a6e | 2025-01-10 20:19:03 | Fix indent. | |
|
4e5e13a2 | 2025-01-09 16:47:24 | Run TCP sysctl ident and drop with shared net lock. Convert exclusive net lock for TCPCTL_IDENT and TCPCTL_DROP to shared net lock and push it down into tcp_ident(). Grab the socket lock there with in_pcbsolock_ref(). Move socket release from in_pcbsolock() to in_pcbsounlock_rele() and add _ref and _rele suffix to the inpcb socket lock functions. They both lock and refcount now. in_pcbsounlock_rele() ignores NULL sockets to make the unlock path in error case simpler. Socket lock also protects tcp_drop() and tcp_close() now, so the socket pointer from incpb may be NULL during unlock. In tcp_ident() improve consistency check of address family. OK mvs@ | |
|
476a4da7 | 2025-01-05 12:36:48 | Retire PR_MPSOCKET flag. TCP socket layer is MP safe for more than a week now. That means all protocols with pr_usrreqs have the PR_MPSOCKET flag. Remove PR_MPSOCKET and use the logic that was used when set. OK mvs@ | |
|
9b315513 | 2025-01-05 12:23:38 | TCP integer sysctl variables are all atomic. Remove net lock. OK mvs@ | |
|
8fed5aa8 | 2025-01-05 12:18:48 | Read tcp_do_ecn only once during packet processing and mark atomic. OK mvs@ | |
|
b869693b | 2025-01-05 12:10:39 | Reorder struct inpcb fields and shorten struct inpcb_iterator. OK mvs@ | |
|
ab8da1a7 | 2025-01-04 15:57:02 | Relax sockets splicing locking. Sockets splicing works around sockets buffers which have their own locks for all socket types, especially sblock() on `so_snd' which keeps sockets being spliced. - sosplice() does read-only sockets options and state checks, the only modification is `so_sp' assignment. The SB_SPLICE bit modification, `ssp_socket' and `ssp_soback' assignment protected with `sb_mtx' mutex(9). PCB layer does corresponding checks with `sb_mtx' held, so shared solock() is pretty enough in sosplice() path. Introduce special sosplice_solock_pair() for that purpose. - sounsplice() requires shared socket lock only around so{r,w}wakeup calls. - Push exclusive solock() down to tcp(4) case of somove(). Such sockets are not ready do unlocked somove() yet. ok bluhm | |
|
84d9c64a | 2025-01-03 21:27:40 | Use atomic operations to modify the MTU of route. When unlocking TCP, path MTU discovery will run in parallel. To keep route MTU consistent, make access to rt_mtu atomic. Use compare-and-swap function to detect whether another thread is modifying the MTU field. In this case skip updating rt_mtu. OK mvs@ | |
|
4ab10cec | 2025-01-03 17:23:51 | Reference count the inpcb in TCP timers. Switch from struct tcpcb to inpcb in the TCP timer argument. The latter already has a reference counter. Increment it at timeout_add() and decrement at timeout_del() or when handler runs. The reaper timeout is special as it does not need a reference, the inpcb is already dead. Use special field t_timer_reaper instead of regular TCP timeout and run it without reference or lock. OK mvs@ | |
|
268735c4 | 2025-01-03 12:56:14 | Do not unlock socket within sorele(). Unlock it outside if required. This time socket could be protected by different locks, include different shared solock() variations. sorele() does nothing that required to lock socket, no reason to release it locked. ok bluhm | |
|
27fdb19e | 2025-01-03 00:49:26 | Run TCP SYN cache timer with shared net lock. Reset the inp_socket pointer in in_pcbdetach() when a socket is about to be freed. By protecting this with a mutex, it is possible to reliable retrieve the socket from an inpcb reference. By using the socket refcount of the listen socket, switch from mutex to socket lock in syn_cache_timer(). tested by Hrvoje Popovski; OK mvs@ | |
|
3064edb0 | 2025-01-02 10:55:18 | Reference count the listen inpcb in the TCP SYN cache. To make progress in unlocking TCP input path, more reference counting is needed. The SYN cache has a reference to the listen socket in form of a struct tcpcb. Instead of adding a refcount to tcpcb, it is easier to use a struct inpcb pointer which already has a refcount. Acquire and hold the reference while running SYN cache timer. OK mvs@ | |
|
66570633 | 2025-01-01 13:44:22 | Fix whitespace. | |
|
507b5b41 | 2024-12-31 12:19:46 | Use per-sockbuf mutex(9) to protect `so_snd' buffer of tcp(4) sockets. Even for tcp(4) case, sosend() only checks `so_snd' free space and sleeps if necessary, actual buffer handling happens in solock()ed PCB layer. Only unlock sosend() path, the somove() is still locked exclusively. The "if (dosolock)" dances are useless, but intentionally left as is. Tested and ok by bluhm. | |
|
77957d73 | 2024-12-30 12:20:39 | Remove net lock from TCP syn cache sysctl. TCP syn cache is protected by mutex. Make access to its sysctl variables either atomic or put them into this mutex. Then net lock can be removed. OK mvs@ | |
|
f9d292df | 2024-12-28 22:17:09 | Read more TCP sysctl variables atomically. OK mvs@ | |
|
7c6c9ed7 | 2024-12-27 10:15:09 | Unlock ah_sysctl() and ipcomp_sysctl(). Both are atomically accessed `ah_enable' and `ipcomp_enable' booleans and per-CPU counters based statistics. esp_sysctl() is much more system wide, so unlock it separately. ok bluhm | |
|
20e4d86e | 2024-12-26 12:16:17 | Declare TCP variable tcprexmtthresh const. OK mvs@ | |
|
cc25bede | 2024-12-26 10:24:57 | Run TCP output in parallel. When called with shared netlock together with socket lock, tcp_output() is MP safe. This is the lock for tcpcb. Mark TCP protocol with PR_MPSOCKET. Also t_oobflags is protected this way, allowing parallel pru_rcvoob(). OK mvs@ | |
|
535d4cde | 2024-12-26 10:15:27 | Make access to tcp_mssdflt atomic. To further unlock TCP sysctl, we need atomic access to tcp_mssdflt. pf(4) is reading the value multiple times. Better read it once and pass mssdflt down the call stack. In pf_calc_mss() was a potential integer underflow. Use the signed variant imax(9) and imin(9) like it has been fixed it in TCP stack. OK mvs@ | |
|
ac935c19 | 2024-12-24 19:16:53 | Protect send buffer with mutex in tcp_mss_update(). To prepare for running tcp_output() in parallel, protect socket send buffer high water field and sbreserve() with mutex. OK mvs@ | |
|
8d117ff8 | 2024-12-24 16:27:07 | Put soassertlocked() into in_pcbrtentry(). Assert that socket lock has been taken when modifying the route cache at the inpcb. This is stronger than only documenting inp_route with [s]. OK mvs@ | |
|
058de743 | 2024-12-24 12:22:17 | Make tcp_mss() MP safe so that it could be called with socket lock. Read tcp_mssdflt and tcp_do_rfc3390 atomically. Address family must be AF_INET or AF_INET6, panic if not. This makes goto out a bit simpler. Route MTU must be read once, another thread might modify it. Fix the signed versus unsigned comparison with max() and min(). OK mvs@ | |
|
8ee0399b | 2024-12-21 00:10:04 | Kill `inp_notify' list remains. This was the list where PCBs were temporary linked to avoid sleep with `inpt_mtx' mutex(9) held. in_pcbnotifyall() and in6_pcbnotify are the last list users, so switch them to in_pcb_iterator() too, moreover they already do in_pcb_is_iterator() check. Note, in_pcb_iterator() does necessary reference couter handling, unlocked `inp' dereference is safe. ok bluhm | |
|
7aa20da4 | 2024-12-20 21:30:17 | Declare some global TCP variables constant. OK mvs@ | |
|
3458005d | 2024-12-20 19:20:34 | Fix signed integer comparison in tcp mss. In tcp_mss_adv() max(9) was used to guarantee that mss it not too small. Unfortunately max() uses u_int and mss could get negative in some error conditions. Rearrange the code to directly return in case of errors. Also read tcp_mssdflt only once to head towards atomic integer sysctl. OK mvs@ | |
|
f1bf6f4e | 2024-12-19 22:11:35 | Use per-sockbuf mutex(9) to protect `so_rcv' buffer of tcp(4) sockets. Only unlock soreceive() path, somove() path still locked exclusively. Also exclusive socket lock will be taken in the soreceive() path each time before pru_rcvd() call. Note, both socket and `sb_mtx' locks are held while SS_CANTRCVMORE modified, so socket lock is enough to check it in the protocol input path. ok bluhm | |
|
10d5b13e | 2024-12-19 22:10:35 | Unlock carp_sysctl(). This is the statistics implemented with per-CPU counters and the `carp_opts' array of atomically accessed integers. Replace the `carp_opts' array with individual `carpctl_*' variables and use sysctl_bounded_arr() instead of sysctl_int(). Keep current `carpctl_*' variables bounds as is, they would be adjusted with separate diff. ok bluhm | |
|
f02897e8 | 2024-12-07 01:14:45 | Embed scope-id when sending NA. Also, do it when the link state is up because the routing entry to the mutlicast address is not ready yet when the carp becomes master. ok florian | |
|
dac6e075 | 2024-12-04 22:48:41 | Push locking down to udp_sysctl(). No locks required for per-CPU counters based protocol statistics. Atomically accessed `udpctl_vars' variables are already moved from the net lock, sysctl(2) related locks are useless for them. Complicated UDPCTL_BADDYNAMIC and UDPCTL_ROOTONLY cases were left as is. ok bluhm | |
|
854a7367 | 2024-12-04 22:24:11 | Push locking down to icmp_sysctl(). Keep locking only for ICMPCTL_REDIRTIMEOUT case. It is complicated, so left it as is. ICMPCTL_STATS loads per-CPU counters into local data, so no locking required. `icmpctl_vars' are atomically accessed integers. Except `icmperrppslim' they are simply booleans, so nothing special required. Used the local `icmperrppslim_local' variable to load `icmperrppslim' value because it it could have negative values. claudio@ proposed to always load such values to local variables, so I want to try this notation. ok bluhm | |
|
b9b60940 | 2024-12-04 18:20:46 | Unlock gre_sysctl(). Both `gre_allow' and `gre_wccp' are atomically accessed integers. They could have only '0' and '1' values, so no extra dances around atomic_load_int(9) required. ok bluhm | |
|
c77b3186 | 2024-11-21 20:15:44 | During IP deliver, propagate correct address family. If ip_deliver() changes the af, e.g. for ip6 over ip4, then it cannot rely on the caller ip_ours() to enqueue the packet in the correct queue. ip_deliver() has to do the enqueue depending on the changed af. from markus@ | |
|
0b21866c | 2024-11-08 21:40:39 | Use mutex of receive socket buffer to protect so_oobmark. Socket field so_oobmark belongs to receive path, so use so_rcv mutex to protect it. Although tcp_input() is still exclusively locked, put mutex there to prepare further unlocking. OK mvs@ | |
|
2162e93b | 2024-11-08 15:46:55 | TCP send and receive space update are MP safe. tcp_update_sndspace() and tcp_update_rcvspace() only read global variables that do not change after initialization. Mark them as such. Add braces around multi-line if blocks. ok mvs@ | |
|
ea7e0400 | 2024-11-08 10:24:13 | Use PCB iterator for raw IPv6 input loop. Implement inpcb iterator in rip6_input(). Factor out the real work to rip6_sbappend(). Now UDP broadcast and multicast, raw IPv4 and IPv6 input work similar. While there, make rip_input() look more like rip6_input(). OK mvs@ | |
|
7f618044 | 2024-11-05 22:44:20 | Use PCB iterator for raw IP input deliver loop. Inspired by mvs@ idea of the iterator in the UDP multicast loop, implement the same for raw IP input delivery. This removes an unneccesary rwlock and only uses table mutex. When comparing the inp routing table, address and port, the table lock must be held. So assume that in_pcb_iterator() already has the table mutex and hold it while traversing the list and doing the checks. Release the mutex during mbuf copy, socket buffer append and the upcalls. Adapt the logic for both rip_input() and udp_input(). In rip_input() move the actual work to rip_sbappend(). This can be called without mutex during list traversal and for the final element. OK mvs@ | |
|
f11c1ce4 | 2024-11-05 10:49:23 | Replace rwlock with iterator in UDP input multicast loop. The broadcast and multicast loop in udp_input() is protected by the table mutex. The relevant PCBs were collected in a separate list, which was processed while the table notify rwlock was held. When sending UDP multicast packets over vxlan(4) configured over UDP with multicast groups, this lock was taken recursively causing a kernel crash. By using an iterator, traversing the PCB list of the table does not require to hold the mutex all the time. Only while accessing the next element after the iterator, the mutex is taken for a short time. udp_sbappend() and the upcall to vxlan_input() is done with neither mutex nor rwlock. The PCB is reference counted while traversing the list. crash reported by Holger Glaess; iterator implemented by mvs@; tested and fixed by bluhm@; OK mvs@ | |
|
05892b44 | 2024-11-03 14:28:06 | Clear UDP checksum out flag when stripping UDP header. Some network interfaces, like lo(4) or vio(4), set the M_UDP_CSUM_OUT flag on incoming packets. For optimization they produce packets with M_UDP_CSUM_IN_OK, but the actual checksum field in the packet is wrong. If such a packet is forwarded, the checksum must be calculated. So they also set M_UDP_CSUM_OUT. For protocols tunneled in UDP, udp_input() removes the header, but the mbuf flags stay. This means later processing of the packet may insert an UDP checksum, although it is not UDP anymore. This has been observed when forwarding ping packets between two vxlan(4) interfaces. Then an UDP checksum was inserted into the ICMP packet. Clearing the M_UDP_CSUM_OUT flag when the UDP header is stripped, fixes the problem. OK mvs@ | |
 |
9b75ea9e | 2024-10-13 02:27:44 | remove unneeded timeout.h include | |
|
60f67f01 | 2024-08-26 13:55:14 | Rearrange #ifdef TCP_SIGNATURE to keep braces balanced. | |
|
7af0f08f | 2024-08-22 10:58:31 | Unlock unlock ipip_sysctl(). - IPIPCTL_ALLOW - atomically accessed integer; - IPIPCTL_STATS - per-CPU counters; In ipip_input() load `ipip_allow' value to `ipip_allow_local' and pass it down to ipip_input_if() as `allow' arg. ok bluhm | |
|
225bc4fe | 2024-08-21 12:53:36 | Revert previous. It was committed mistakenly. | |
|
9c33e071 | 2024-08-21 07:36:12 | Unlock ipip_sysctl(). - IPIPCTL_ALLOW - atomically accessed integer; - IPIPCTL_STATS - per-CPU counters; ok bluhm | |
|
02e922b0 | 2024-08-20 07:47:25 | Unlock etherip_sysctl(). - ETHERIPCTL_ALLOW - atomically accessed integer; - ETHERIPCTL_STATS - per-CPU counters ok bluhm | |
|
572e76f6 | 2024-08-20 07:46:27 | Unlock igmp_sysctl(), pfsync_sysctl() and rip6_sysctl(). Each of them is the only IGMPCTL_STATS, PFSYNCCTL_STATS and RIPV6CTL_STATS per-CPU counters. sysctl_rdstruct() has "newp != NULL" check within and also returns EPERM, no need for redundant check in igmp_sysctl(). ok bluhm | |
|
60728a2f | 2024-08-16 09:20:34 | Introduce PR_MPSYSCTL flag to mark mp-safe (*pr_sysctl)() handlers and unlock both divert_sysctl() and divert6_sysctl(). Unlock them together, because they are identical and pretty simple: - DIVERTCTL_RECVSPACE and DIVERTCTL_SENDSPACE - atomically accessed integers; - DIVERTCTL_STATS - per-CPU counters; ok bluhm | |
|
fc751501 | 2024-08-12 11:25:27 | Run network protocol timer without kernel lock. Mark slow and fast protocol timeouts as MP safe. This means they run on a spearate thread without holding the kernel lock. IGMP and MLD6 cannot run in parallel, they use exclusive net lock to protect themselves. As a performance optimization global variables are used to skip igmp_fasttimo() and mld6_fasttimeo() if no multicast is active. These global variables use atomic operations and memory barriers to work lockless. IPv6 fragment timeout protects itself with a mutex. TCP timers also run without kernel lock now. The whole TCP stack holds exclusive net lock, so additional kernel lock is useless. OK mvs@ | |
|
3e38cd6f | 2024-08-06 20:15:53 | Unlock `udpctl_vars'. `udp_sendspace' and `udp_recvspace' are integers which read-only accessed in udp_attach(). `udpcksum' read-only accessed in udp_output(). No netlock required to modify them through sysctl(2). ok bluhm | |
|
c80589b8 | 2024-08-06 16:56:09 | Unlock sysctl net.inet.ip.directed-broadcast. ip_directedbcast is read once in either ip_input() or pf_test() during packet processing. So writing the variable does not need net lock. OK mvs@ | |
|
59d5251a | 2024-07-26 14:38:20 | Run UDP input on multiple CPU in parallel. The socket layer of UDP has been made fully MP safe. UDP output is MP safe for a while. mvs@ has fixed the missing pieces in socket splicing recently. This means that complete UDP stack can be processed by multiple threads now. Activate multi processing for udp_input() when called with IPv4 or IPv6 packets. Usually IP processing runs on multiple softnet threads with shared net lock. From there local packets are queued and processed by one thread with exclusive net lock. If the PR_MPINPUT flag is set, protocol input is called directly from IP input on multiple threads, with shared net lock and no additional queueing. tested by Hrvoje Popovski; OK mvs@ | |
|
134de7a2 | 2024-07-20 17:26:19 | Unlock udp(4) somove(). Socket splicing belongs to sockets buffers. udp(4) sockets are fully switched to fine-grained buffers locks, so use them instead of exclusive solock(). Always schedule somove() thread to run as we do for tcp(4) case. This brings delay to packet processing, but it is comparable wit non splicing case where soreceive() threads are always scheduled. So, now spliced udp(4) sockets rely on sb_lock() of `so_rcv' buffer together with `sb_mtx' mutexes of both buffers. Shared solock() only required around pru_send() call, so the most of somove() thread runs simultaneously with network stack. Also document 'sosplice' structure locking. Feedback, tests and OK from bluhm. | |
|
9c4c1a6f | 2024-07-19 16:58:31 | Unlock sysctl net.inet.ip.redirect and net.inet6.ip6.redirect. Variable ip and ip6 sendredirects is only read once during packet processing. Use atomic_load_int() to access the value in exactly one read instruction. No memory barriers needed as there is no correlation with other values. Sort the ip and ip6 checks, so the difference is easier to see. Move access to global variable to the end. OK mvs@ | |
|
18474754 | 2024-07-19 15:41:58 | Relax socket lock assertion in UDP input and send. OK mvs@ | |
|
1f9e444e | 2024-07-14 18:53:39 | Unlock IPv6 sysctl net.inet6.ip6.forwarding from net lock. Use atomic operations to read ip6_forwarding while processing packets in the network stack. To make clear where actually the router property is needed, use the i_am_router variable based on ip6_forwarding. It already existed in nd6_nbr. Move i_am_router setting up the call stack until all users are independent. The forwarding decisions in pf_test, pf_refragment6, ip6_input do also not interfere. Use a new array ipv6ctl_vars_unlocked to make transition of all the integer sysctls easier. Adapt IPv4 to the new style. OK mvs@ | |
|
4fbed6ad | 2024-07-13 13:20:44 | Add condition to ip_gre.c in files. Use gre condition in conf/files for compiling netinet/ip_gre.c only if needed. Remove #if NGRE > 0 from ip_gre.c that caused ramdisk build to compile an empty C file. OK kn@ deraadt@; input jsg@ | |
|
093b188b | 2024-07-13 12:00:11 | Mark IP protocol GRE as MP safe from socket layer. The pipex code in gre_send() matches more or less what udp_send() does. This has been MP safe for a long time. rip_send() is already called with PR_MPSOCKET. OK mvs@ | |
|
4c2e8f66 | 2024-07-12 19:50:35 | Remove internet PCB mutex. All incpb locking has been converted to socket receive buffer mutex. Per PCB mutex inp_mtx is not needed anymore. Also delete PRU related locking functions. A flag PR_MPSOCKET indicates whether protocol functions support parallel access with per socket rw-lock. TCP is the only protocol that is not MP capable from the socket layer and needs exclusive netlock. OK mvs@ | |
|
2c9f5b1f | 2024-07-12 09:25:27 | Run sysctl net.inet.ip.forwarding without net lock. The places in packet processing where ip_forwarding is evaluated have been consolidated. The remaining pieces in pf test, ip input, and icmp input do not need consistent information. If the integer value is changed by another CPU, it is harmless. The sysctl syscall sets the value atomically, so add atomic read in network processing and remove the net lock in sysctl IPCTL_FORWARDING. OK claudio@ mvs@ | |
|
0e25137a | 2024-07-02 18:33:47 | Read IPsec forwarding information once. Fix MP race between reading ip_forwarding in ip_input() and checking ip_forwarding == 2 in ip_output(). In theory ip_forwarding could be 2 during ip_input() and later 0 in ip_output(). Then a packet would be forwarded that was never allowed. Currently exclusive netlock in sysctl(2) prevents all races. Introduce IP_FORWARDING_IPSEC and pass it with the flags parameter that was introduced for IP_FORWARDING. Instead of calling m_tag_find(), traversing the list, and comparing with NULL, just check the PACKET_TAG_IPSEC_IN_DONE bit. Reading ipsec_in_use in ip_output() is a performance hack that is not necessary. New code only checks tree bits. OK mvs@ | |
|
a96be699 | 2024-06-24 12:19:19 | Explicitly allocate stack memory for ICMP payload in IPv4 forward. Old ip_forward() allocated a fake mbuf copy on the stack to send an ICMP packet after ip_output() has failed. It seems easier to just copy the data onto the stack that icmp_error() may use. Only if the ICMP error packet is acutally sent, create the mbuf. m_dup_pkthdr() uses atomic operation to link the incpb to mbuf. pf_pkt_addr_changed() was immediately called afterwards to remove the linkage again. Also m_tag_delete_chain() was overhead. New code uses less CPU locking in the hot path. OK deraadt@ claudio@ | |
|
ab457133 | 2024-06-20 19:25:42 | Read IPv6 forwarding value only once while processing a packet. IPv4 uses IP_FORWARDING to pass down a consistent value of net.inet.ip.forwarding down the stack. This is needed for unlocking sysctl. Do the same for IPv6. Read ip6_forwarding once in ip6_input_if() and pass down IPV6_FORWARDING as flags to ip6_ours(), ip6_hbhchcheck(), ip6_forward(). Replace the srcrt value with IPV6_REDIRECT flag for consistency with IPv4. To have common syntax with IPv4, use ip6_forwarding == 0 checks instead of !ip6_forwarding. This will also make it easier to implement net.inet6.ip6.forwarding=2 for IPsec only forwarding later. In nd6_ns_input() and nd6_na_input() read ip6_forwarding once and store it in i_am_router. The variable name has been chosen to avoid confusion with is_router, which indicates router flag of the packet. Reading of ip6_forwarding is done independently from ip6_input_if(), consistency does not really matter. One is for ND router behavior the other for forwarding. Again use the ip6_forwarding != 0 check, so when ip6_forwarding IPsec only value 2 gets implemented, it will behave like a router. OK deraadt@ sashan@ florian@ claudio@ | |
|
06c3a42d | 2024-06-20 19:25:04 | Do not send ICMP redirect if IP forwarding is IPsec only. If sysctl net.inet.ip.forwarding is set to 2, only packets processed by IPsec are forwarded. I this case behave more like a router than a host and do not accept ICMP redirect packets. OK deraadt@ sashan@ florian@ claudio@ | |
|
84b2c343 | 2024-06-07 18:24:16 | Read IP forwarding variables only once. Do not assume that ip_forwarding and ip_directedbcast cannot change while processing one packet. Read it once and pass down its value with a flag. This is necessary for unlocking the sysctl path. There are a few places where a consistent value does not really matter, they are unchanged. Use a proper ip_ prefix for the global variable. OK claudio@ | |
|
b6845120 | 2024-06-07 13:15:25 | remove MAXBUFSIZ define, unused since rev 1.33 | |
|
81a70692 | 2024-06-07 08:02:17 | remove unused packet header length defines | |
|
b2d46992 | 2024-05-17 20:44:36 | IPv6 has to use ip6_defhlim, not ip_defttl. OK claudio@ | |
|
a1db6f2d | 2024-05-16 13:01:04 | Fix IPsec in use with IP forwarding 2 logic. If sysctl net.inet.ip.forwarding is 2, only packets processed by IPsec are forwarded. Variable ipsec_in_use is a shortcut to avoid IPsec processing if no policy has been configured. With ipsec_in_use unset and ipforwarding set to IPsec only, the packet must be dropped. OK claudio@ | |
|
3ceb73ad | 2024-05-14 09:39:02 | Sanity check for TSO payload length in TCP chopper. Although it should not happen, check that ph_mss is not 0 in tcp_chopper(). This could catch errors in the LRO path of network drivers. Better count bad packet and drop it rather than ending in an endless loop. The new logic is analog to a recent change in the hardware TSO path in the drivers. OK jan@ | |
|
0f9e9ec2 | 2024-05-13 01:15:50 | remove prototypes with no matching function ok mpi@ | |
|
652f3531 | 2024-05-08 13:01:30 | Fix route leak in ip input. In previous commit when refactoring the route cache, a rtfree() has been forgotten. For each forwarded packet the reference counter of the route entry was increased. This eventually leads to an integer overflow and triggers kassert. reported by and OK jan@ | |
|
5bda38b8 | 2024-04-19 10:13:58 | Merge IPv4 and IPv6 options in inpcb. A internet PCB has either inp_options or inp_outputopts6. Put them into a common anonymous union. OK mvs@ kn@ | |
|
ace0f189 | 2024-04-17 20:48:51 | Use struct ipsec_level within inpcb. Instead of passing around u_char[4], introduce struct ipsec_level that contains 4 ipsec levels. This provides better type safety. The embedding struct inpcb is globally visible for netstat(1), so put struct ipsec_level outside of #ifdef _KERNEL. OK deraadt@ mvs@ | |
|
00dd3069 | 2024-04-16 12:56:39 | Use route cache function in IP input. Instaed of passing a struct rtentry from ip_input() to ip_forward() and then embed it into a struct route for ip_output(), start with struct route and pass it along. Then the route cache is used consistently. Also the route cache hit and missed counters should reflect reality after this commit. There is a small difference in the code. in_ouraddr() checks for NULL and not rtisvalid(). Previous discussion showed that the route RTF_UP flag should only be considered for multipath routing. Otherwise it does not mean anything. Especially the local and broadcast check in in_ouraddr() should not be affected by interface link status. When doing cache lookups, route must be valid, but after rtalloc_mpath() lookup, use any route that route_mpath() returns. OK claudio@ | |
|
667565e9 | 2024-04-15 18:31:04 | Delete unused inp_csumoffset define. OK mvs@ | |
|
7019ae97 | 2024-04-14 20:46:27 | Run raw IP input in parallel. Running raw IPv4 input with shared net lock in parallel is less complex than UDP. Especially there is no socket splicing. New ip_deliver() may run with shared or exclusive net lock. The last parameter indicates the mode. If is is running with shared netlock and encounters a protocol that needs exclusive lock, the packet is queued. Old ip_ours() always queued the packet. Now it calls ip_deliver() with shared net lock, and if that cannot handle the packet completely, the packet is queued and later processed with exclusive net lock. In case of an IPv6 header chain, that switches from shared to exclusive processing, the next protocol and mbuf offset are stored in a mbuf tag. OK mvs@ | |
|
3b9d585e | 2024-04-13 23:44:11 | correct indentation no functional change, found by smatch warnings ok miod@ bluhm@ |