Since the function name is misleading since it is not changing
anything, name it similarly to other cls.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The name cls_bpf_modify_existing is highly misleading, as it indeed does
not modify anything existing. It does not modify at all.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
For check in tcf_exts_dump use tcf_exts_has_actions helper instead
of exts->nr_actions for checking if there are any actions present.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Leave it to tcf_action_exec to return TC_ACT_OK in case there is no
action present.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
These two helpers are doing the same as tcf_exts_has_actions, so remove
them and use tcf_exts_has_actions instead.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Use the tcf_exts_has_actions helper instead or directly testing
exts->nr_actions in tcf_exts_exec.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The rest of the helpers are named tcf_exts_*, so change the name of
the action number helpers to be aligned. While at it, change to inline
functions.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Since tcf_em_tree_validate could be always called on a newly created
filter, there is no need for this change function.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Even if it is only for classid now, use this common struct a be aligned
with the rest of the classful qdiscs.
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This patch fixes the __udivdi3 undefined error reported by
test robot.
Fixes: b8c17f7088 ("net: hns: Add self-adaptive interrupt coalesce support in hns driver")
Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
This was supposed to be a bitwise OR but there is a || vs | typo.
Fixes: 864dc729d5 ("net: phy: marvell: Refactor m88e1121 RGMII delay configuration")
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
Pull sparc fixes from David Miller:
- block interrupts properly across the entire MMU context change (both
the hw MMU context change and the TSB table change) so that we don't
get a perf event interrupt in the middle. From Rob Gardner.
- be sure to register hugepages early enough, from Nitin Gupta.
- UltraSPARC-III user copy exception handling would return garbage for
the copied length in some circumstances.
* git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc:
sparc64: Fix exception handling in UltraSPARC-III memcpy.
sbus: Convert to using %pOF instead of full_name
sparc: defconfig: Cleanup from old Kconfig options
sparc64: Register hugepages during arch init
sparc64: Prevent perf from running during super critical sections
->reencode_message() and last minute RADOS semantic changes in v12.1.2.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAABCAAGBQJZhI6QAAoJEEp/3jgCEfOLeQYH/0b92RFzmsqwPI+U7iXdD06O
r0EXbT5dydMngJkWz/i3jBX8cMBvZyNhBh77VPDYXoFUp8//8uv5w73BkXe8JE08
+gLZU4oP/k7kl/YBYXgCcJYj7eIBFzqNvsWurKKHY/X3xrvEZ0HT+oub92xOUgRM
IBnZb1gZ4TJQT1MxqKOwb5aqcxaXlrOGfX7Di0aU3PFQXj5VnBI25NUQF1bgd9+A
MbhHpob6cbWZWzVdf0fTl28q9pStq4qggevRSM/5ZH/bETO8C80XYTuaPoLcQ0pY
VfpwgWIAPwotw9KU7W+ane13BURw76+pWMHaUZgiJKRyuRMBOT/gaER+AUeuR1o=
=XH7K
-----END PGP SIGNATURE-----
Merge tag 'ceph-for-4.13-rc4' of git://github.com/ceph/ceph-client
Pull ceph fixes from Ilya Dryomov:
"A bunch of fixes and follow-ups for -rc1 Luminous patches: issues with
->reencode_message() and last minute RADOS semantic changes in
v12.1.2"
* tag 'ceph-for-4.13-rc4' of git://github.com/ceph/ceph-client:
libceph: make RECOVERY_DELETES feature create a new interval
libceph: upmap semantic changes
crush: assume weight_set != null imples weight_set_size > 0
libceph: fallback for when there isn't a pool-specific choose_arg
libceph: don't call ->reencode_message() more than once per message
libceph: make encode_request_*() work with r_mempool requests
Now we hit the usual ASoC-fix-flood in the middle of release.
Most of changes are trivial and device-specific, while one significant
change is the fix for unbalanced of_graph_*() refcounts. This
involved with the change in the graph API itself that had been a bit
messy.
-----BEGIN PGP SIGNATURE-----
iQJCBAABCAAsFiEECxfAB4MH3rD5mfB6bDGAVD0pKaQFAlmESFAOHHRpd2FpQHN1
c2UuZGUACgkQbDGAVD0pKaQ6ow/6A2PhAU8+WfUR0d/EWwJ5Gd+iEnp79jEIvPmX
1tqBFRhft7yVUxK7TDQ0HSQCktDKbmHszUxOga8EU9j6BjNyrvUdr3OY7X8kSCbP
kpVQ9R/sHd/0iRDA8Cn+PazO6io3PpqKAvlrMf5p9uSZ+Ic6+MCQAUb5dHgqjOQq
kLGIgyOxoziKpMkb7SuLGbxFTJiioe0HFyMSPTHHC5jB4B0pyBhMaIUXlKypnOdk
IHt68GuhNG+k20jAqlfRV/HCKPTgMvlZAcU+KNB1KqUpcjTy53Myi1yQ4A/ngfgj
rbEObaluJF1G+3texjMRatpzsBwNOitrdtK5yqTncXyrbOR4YUSFxdM0FF7jcvzn
tnHpZTCNOAvosQ3K8AuFy4j+kypYEB3MycPMGOFk+BwslFgBME6vBzKyxzGann7y
d+2imj7NC/uPor5qpAqw01xpsg57QKOGCzPWO3r/rZg7u20ejiJjVFhRH6hKnS7W
NfTpug8O1aedvzOSChYid7ujFrHZhscwDLvqn1qKce5za68cHSNpTDxG0RkwRwAD
fBzHTi3yEQKt74Mfb2K7MaY9g9W/oFpNIQBnNyrTYWh8AyumjUoBpq34MiJl5iMm
ZV64z7r79ANkE31qNKG8EXLDR8LEymGj2JfMdDuBiVyXkjjEulCzY5KIA5rxQJfG
X+4GZmg=
=k+0o
-----END PGP SIGNATURE-----
Merge tag 'sound-4.13-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
Pull sound fixes from Takashi Iwai:
"Now we hit the usual ASoC-fix-flood in the middle of release.
Most of the changes are trivial and device-specific, while one
significant change is the fix for unbalanced of_graph_*() refcounts.
This involved a change in the graph API itself that had been a bit
messy"
* tag 'sound-4.13-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound:
ALSA: hda - Fix speaker output from VAIO VPCL14M1R
device property: Fix usecount for of_graph_get_port_parent()
ASoC: rt5665: fix wrong register for bclk ratio control
ASoC: Intel: Use MCLK instead of BLCK as the sysclock for RT5514 codec on kabylake platform
ASoC: Intel: Enabling ASRC for RT5663 codec on kabylake platform
ASoC: codecs: msm8916-analog: fix DIG_CLK_CTL_RXD3_CLK_EN define
ASoC: Intel: Skylake: Fix missing sentinels in sst_acpi_mach
ASoC: sh: hac: add missing "int ret"
ASoC: samsung: odroid: Fix EPLL frequency values
ASoC: sgtl5000: Use snd_soc_kcontrol_codec()
ASoC: rt5665: fix GPIO6 pin function define
ASoC: ux500: Restore platform DAI assignments
ASoC: fix pcm-creation regression
ASoC: do not close shared backend dailink
ASoC: pxa: SND_PXA2XX_SOC should depend on HAS_DMA
ASoC: Intel: Skylake: Fix default dma_buffer_size
ASoC: rt5663: Update the HW default values based on the shipping version
ASoC: imx-ssi: add check on platform_get_irq return value
Including:
- Fix a scheduling-while-atomic bug in the AMD IOMMU driver. It
was found after the checker was enabled earlier.
- A fix for the virtual APIC code in the AMD IOMMU driver which
delivers device interrupts directly into KVM guests for
assigned devices.
- Fixes for the recently merged lock-less page-table code for
ARM. The redundant TLB syncs got reverted and locks added
again around the TLB sync code.
- Fix for error handling in arm_smmu_add_device()
- Address sanitization fix for arm io-pgtable code
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAABAgAGBQJZhDD/AAoJECvwRC2XARrj3YkP/2+yZgffkQj9Xp2izSMgQYyI
VqqYj2/oaolcbYqY5iv6OqU/W2soutU9x6Qf/PlJ4aHD+iQMlRwKqWwkiI/k5JKA
7t1qZwzHucygBgrckjZBtNhU8sfGsCPDeu1Vw6o2rLHQvYUBEI6Hk/jS//YKIhrt
453N2z/ABXtjFHYVLtYPrJBvSrhnKHcR55enT1AQL1nKYhF3ZqlDwiv28P6S4JhJ
i3OWX8S2jy59Jw3H5FggNBAP4oTN3PrLdnR9EzIB7kkWXC0CJgOmlFiRfNTrh25U
+GPXL6NHEt2EuqP+qjH2UV+Lb3IY+uekSusDLemMTIVh1Zs5U/cm5cglHZtaWzBZ
tt+ZsGuErFAaJt26ubWOk96+9WQchgfQVedAoIise7sT6OlThQUPba7pyHAiA2/X
umm8Fu/78cNDLQNssWNC87xvpV3TqwJVceXqIRmw8oRwTWzMfkkerOZzDhyGO/am
UqyulMX7QQrBygmgSzd4qZvOVqRBGO2caSVdsvmrSzYeRXxBlnMrdv+gTxt65Wju
m6MkWuFym6CbvWdHg9nUSwuX4eOIP0PpME2zXMlg7DMFJ8AJoABxLdar1cCEOKGi
6Vq5hHQQ8pGKDIVw5Cz3YC45Fg2XOr6RjsTiDZf8gYJsEqhDHZ8XYnZ1GaFsjvVX
+653QdN/pqWCdukUC4jc
=Z9oV
-----END PGP SIGNATURE-----
Merge tag 'iommu-fixes-v4.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu
Pull IOMMU fixes from Joerg Roedel:
- fix a scheduling-while-atomic bug in the AMD IOMMU driver. It was
found after the checker was enabled earlier.
- a fix for the virtual APIC code in the AMD IOMMU driver which
delivers device interrupts directly into KVM guests for assigned
devices.
- fixes for the recently merged lock-less page-table code for ARM. The
redundant TLB syncs got reverted and locks added again around the TLB
sync code.
- fix for error handling in arm_smmu_add_device()
- address sanitization fix for arm io-pgtable code
* tag 'iommu-fixes-v4.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu:
iommu/amd: Fix schedule-while-atomic BUG in initialization code
iommu/amd: Enable ga_log_intr when enabling guest_mode
iommu/io-pgtable: Sanitise map/unmap addresses
iommu/arm-smmu: Fix the error path in arm_smmu_add_device
Revert "iommu/io-pgtable: Avoid redundant TLB syncs"
iommu/mtk: Avoid redundant TLB syncs locally
iommu/arm-smmu: Reintroduce locking around TLB sync operations
- Fix NULL pointer dereference for block I/O during hotplug
MMC host:
- sdhci-of-at91: Fix card detect for non-removable cards
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAABAgAGBQJZhC3oAAoJEP4mhCVzWIwpaAEP+wbSa4FrPIdpMMy8VWtrvD/m
K9GkfcrCbObE7Cmdff7Hy2OhhqNsWBtE0oxsfgf5ynneIGcwXILQAqmNcGEWlHUO
8uR5hI4+Y+ErR6YoNrupPw9rkWeaSiMyeZwlG3DkK1mrdc4P5R6m2UuOPexMWdGS
B9rXsCYHa7xKFAgcjaCMbljEUBCfItEEy/JJ7scly4KOJ7MBrGagcS0wYJBXilD+
DMU1ycjxDYQWjwMzY7WPb9N5iu6Juxpj8Tp7vQDHOWSHb1P7HZJ02dkBvDRf63Cz
bYpsCrmgGrddo+zSDAoGRiElOzYZPiLU8zT7TuvcVvXrXx1a1u5zAFzbrmasiOpi
ChmRn8wBrJTkpdIk38jHpq7eyxPRl21Kif07PL916tVpG6cRqKEozNJiWx8h8SI3
H1uG/MMKqnhygzkG1ComgBxoncE+ZVlTBnn85AgM9M1loYdOLeGKhjFIhFFEITWM
amKodARXWpuO21nfd+86sH/9z0rFEQYUlkcJKU8pMRE4ZEWZ70vyP4U9g8RdGR+I
nC36393gvjAB3a6TUvcwzjyRdiHLWKxSYarZ88EB2ezqlhnqIiMm49B7/0sBELUd
wc6qjRoSLoLzn5Hm6EzmPaYgJ6pBhIVelEnJbNkmM+3aFfWeRewbSnCXG9qq89oa
HuY5mse4UVMOlMRUGLWC
=VpnQ
-----END PGP SIGNATURE-----
Merge tag 'mmc-v4.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
Pull MMC fixes from Ulf Hansson:
"A couple of mmc fixes intended for v4.13-rc4.
MMC core:
- Fix NULL pointer dereference for block I/O during hotplug
MMC host:
- sdhci-of-at91: Fix card detect for non-removable cards"
* tag 'mmc-v4.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc:
mmc: block: bypass the queue even if usage is present for hotplug
mmc: sdhci-of-at91: force card detect value for non removable devices
-----BEGIN PGP SIGNATURE-----
iQIcBAABAgAGBQJZhBe5AAoJEAx081l5xIa+NR8P/02fl5JHVXaPpSvR+VmFzaCD
05yYdc56QhWv81lUWAHEP+pFJTMv49y8qyekqDdEDI4CCP8uU+Sc4zPQ/BgW/62c
9Yqy7RwigEp8FqqttEb3s6RW2OnLTfgXnwvLzXRDyMUkUUBOtL545lgOG/Vrz3Id
c+uRrNXlxYuSfoSjoiqP6VsBXXEYI0eZS9oWWlnTsGAvCI3JK/pSTfj9m9nfdXZ/
ByghEEB+PE59m2nP/egp5CJd/3fJ5GVXqK7ROObpz1Hkd7dSZjVALtLE+KB3bUuV
OvCfStGwfEHs3lMmjxahIn4fBWOeBh0OtQd3j36It7i0aeTPVpXH+o/O1EseT7Ka
gQjr7miBGlZPSJ+EkqObG3Dv2EL3/2S8dDVOLaXbtHXsZHIebumDJ3ogDNq0xPpU
mZjtNBvlDbwzmm7dSw6r9rHZBUzCfCEefmYkVS4tVBl2vMoXQJA6FrYGtgVV2eZO
xUNEgTcmLqMcP38q9/UWC0ZBOQJLoEY5SfVVaLzIIKqHSBJjAbVJXmrrIuVSmnFT
Kie0eBOtMcsToJh7yUR+vvTAByZg4bl8j3hZlX0PtyRwxpavzMvBiHYa4gE2CpU3
EwU7JjqF346CdMvgiRZHpBZXKnGKSrVSDtdnlFrR5C5H7XWTJO3rRPKexcyrCc1F
bNo/diEu4KmywYrXA1tT
=nQKg
-----END PGP SIGNATURE-----
Merge tag 'drm-fixes-for-v4.13-rc4' of git://people.freedesktop.org/~airlied/linux
Pull drm fixes from Dave Airlie:
"Either my email ate everything or everyone is on holidays, either way
all I can find is some lonely AMD fixes"
[ Europe might be on vacation, and the Pacific NW is too hot for work. ]
* tag 'drm-fixes-for-v4.13-rc4' of git://people.freedesktop.org/~airlied/linux:
drm/amdgpu: Use list_del_init in amdgpu_mn_unregister
drm/amdgpu: Fix undue fallthroughs in golden registers initialization
drm/amdgpu: fix header on gfx9 clear state
Fixes for recently merged code:
- a fix for the _PAGE_DEVMAP support, which was breaking KVM on Power9 radix
- avoid a (harmless) lockdep warning in the early SMP code
- return failure for some uses of dma_set_mask() rather than falling back to 32-bits
- fix stack setup in watchdog soft_nmi_common() to use emergency stack
- fix of_irq_to_resource() error check in of_fsl_spi_probe()
Two fixes going to stable:
- fix saving of Transactional Memory SPRs in core dump
- fix __check_irq_replay missing decrementer interrupt
And two misc:
- fix 64-bit boot wrapper build with non-biarch compiler
- work around a POWER9 PMU hang after state-loss idle
Thanks to:
Alistair Popple, Aneesh Kumar K.V, Cyril Bur, Gustavo Romero, Jose Ricardo
Ziviani, Laurent Vivier, Nicholas Piggin, Oliver O'Halloran, Sergei Shtylyov,
Suraj Jitindar Singh, Thomas Gleixner.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAABAgAGBQJZhEH/AAoJEFHr6jzI4aWA8GoP+wdtaBYZIElFMnPay/BY8lSQ
MODF6FlaexdNuVFo5In2BnMAs12bIXDO69wMziqc+uZIA+vI3K7lqsi+8p1R38Ts
FG5OFLzX/gx1ykoeDe0mC8GPUHGVBdMr6+rHzt/9M7PrQWApCoBJLakYIElOrN5B
EiAxunwJQISKNQYYmyVhjMVFq8ydMHSgtR7JVDHTh1AoStaO9M5hoJegD4Dnytaf
F0/0Y5U+NGGWwPIyDrjB4wE9L8NCk0JfvpBhjORMkFSpAOe4VhxZrAGDyaCzVKVh
YkxSirVKuVwEmvV2prmq4SAJi61EMUEPx2WL7hRN3ol/dF+2lU9nl2lyL3yRnGPY
9mtNHur07qoMqyXI8750ayTzbMG75aakJF2be9G+zwNc3Sw8dqJvs2PwpSVdnima
jWmWnRikws4qmEOEBV6nxtWf8qYACHevyK7YzIHDU8SVXLjAkX/u6LZGHAfTQ8Xo
jYcISpiA2ko8YnMR7yy9FZqs4KGqeOotMamgJRGUcSY/9Xn8WDIrtQ4tIuJtGjzW
XFOFafZOzyt6s1dNz01C4hzP60J+/CxPNRxx6zWnbijY6puADu/Mv1g+ai3nYdKA
e/CYt6O8tzooFHCHtVLUPgWA82heDwNtjmlnSikWTdT07KLRhP+0w9Qq8gBzztKT
7B/vPMjCMPQaJrrnj6GX
=Rdy8
-----END PGP SIGNATURE-----
Merge tag 'powerpc-4.13-5' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux
Pull powerpc fixes from Michael Ellerman:
"Fixes for recently merged code:
- a fix for the _PAGE_DEVMAP support, which was breaking KVM on
Power9 radix
- avoid a (harmless) lockdep warning in the early SMP code
- return failure for some uses of dma_set_mask() rather than falling
back to 32-bits
- fix stack setup in watchdog soft_nmi_common() to use emergency
stack
- fix of_irq_to_resource() error check in of_fsl_spi_probe()
Two fixes going to stable:
- fix saving of Transactional Memory SPRs in core dump
- fix __check_irq_replay missing decrementer interrupt
And two misc:
- fix 64-bit boot wrapper build with non-biarch compiler
- work around a POWER9 PMU hang after state-loss idle
Thanks to: Alistair Popple, Aneesh Kumar K.V, Cyril Bur, Gustavo
Romero, Jose Ricardo Ziviani, Laurent Vivier, Nicholas Piggin, Oliver
O'Halloran, Sergei Shtylyov, Suraj Jitindar Singh, Thomas Gleixner"
* tag 'powerpc-4.13-5' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
powerpc/64: Fix __check_irq_replay missing decrementer interrupt
powerpc/perf: POWER9 PMU stops after idle workaround
powerpc/83xx/mpc832x_rdb: fix of_irq_to_resource() error check
powerpc/64s: Fix stack setup in watchdog soft_nmi_common()
powerpc/powernv/pci: Return failure for some uses of dma_set_mask()
powerpc/boot: Fix 64-bit boot wrapper build with non-biarch compiler
powerpc/smp: Call smp_ops->setup_cpu() directly on the boot CPU
powerpc/tm: Fix saving of TM SPRs in core dump
powerpc/mm: Fix pmd/pte_devmap() on non-leaf entries
Mikael Pettersson reported that some test programs in the strace-4.18
testsuite cause an OOPS.
After some debugging it turns out that garbage values are returned
when an exception occurs, causing the fixup memset() to be run with
bogus arguments.
The problem is that two of the exception handler stubs write the
successfully copied length into the wrong register.
Fixes: ee841d0aff ("sparc64: Convert U3copy_{from,to}_user to accurate exception reporting.")
Reported-by: Mikael Pettersson <mikpelinux@gmail.com>
Tested-by: Mikael Pettersson <mikpelinux@gmail.com>
Reviewed-by: Sam Ravnborg <sam@ravnborg.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
The bitmask used to define these values produces overflow, as seen by
this compiler warning:
arch/arm64/kernel/head.S:47:8: warning:
integer overflow in preprocessor expression
#elif (PAGE_OFFSET & 0x1fffff) != 0
^~~~~~~~~~~
arch/arm64/include/asm/memory.h:52:46: note:
expanded from macro 'PAGE_OFFSET'
#define PAGE_OFFSET (UL(0xffffffffffffffff) << (VA_BITS -
1))
~~~~~~~~~~~~~~~~~~ ^
It would be preferrable to use GENMASK_ULL() instead, but it's not set
up to be used from assembly (the UL() macro token pastes UL suffixes
when not included in assembly sources).
Suggested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Suggested-by: Yury Norov <ynorov@caviumnetworks.com>
Suggested-by: Matthias Kaehlcke <mka@chromium.org>
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
In a system with DBM (dirty bit management) capable agents there is a
possible race between a CPU executing ptep_set_access_flags() (maybe
non-DBM capable) and a hardware update of the dirty state (clearing of
PTE_RDONLY). The scenario:
a) the pte is writable (PTE_WRITE set), clean (PTE_RDONLY set) and old
(PTE_AF clear)
b) ptep_set_access_flags() is called as a result of a read access and it
needs to set the pte to writable, clean and young (PTE_AF set)
c) a DBM-capable agent, as a result of a different write access, is
marking the entry as young (setting PTE_AF) and dirty (clearing
PTE_RDONLY)
The current ptep_set_access_flags() implementation would set the
PTE_RDONLY bit in the resulting value overriding the DBM update and
losing the dirty state.
This patch fixes such race by setting PTE_RDONLY to the most permissive
(lowest value) of the current entry and the new one.
Fixes: 66dbd6e61a ("arm64: Implement ptep_set_access_flags() for hardware AF/DBM")
Cc: Will Deacon <will.deacon@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Steve Capper <steve.capper@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
They should be used only when an actual
remote-endpoint is connected.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAABAgAGBQJZectgAAoJEGFBu2jqvgRNZdkP/27ovOMGuzleyl18PiLotyXn
98qxZvnFYyEw258vkUJj0N7Rmf3gMnrfbrRIOGXyOIBpZNBkgEqjGlagXn9Z+o8F
QOERxzDT5JiM6BbR+tRPOmKc07uuccVskOCoYOZ1mk4c336b/m9dNppdWGGY83LC
WJ0OeoFP8k8gUMYqSyKscXyfryP82KbryZH5s5tmSVAnEHf6psUzAUUdQf7An5v2
Up5nD4ljF+leGX6PG8rAFRbMGy/jTHOQi7QafVx3rC+p8SydoUnEWhPTrtkDzRTb
bXtpjSUId7n80qKtyI/vD3ABDBajr4RUzYjuXGfSf32A5uvk7xjzUk8QfwWogxv5
fdOga6ogEOUjgB1L+u2uegHQl3CGSZK5sXufoQivcndBgXqA5S9VkQk/FNjAM9vJ
twyC+s3pgC7cBrHGmM4QFToubYQthmXOevKmKlbRvXyNFHW6BcIPcbq+6t4IDakd
DU7iYRzR+AxVK5fezWOhBxxL2aRK533r/gc1ZOBklRN1gD+5AWLfaQJMNDDdVePJ
8AH+PAZZz2ETXLvMH+D4XSPF6t7gDyZ6Jroh+YJ3i0Y0vOO16Jr2LvJlAClFfQNM
+TRjhaGd4WFrUVnpiyPU9l2MrTOmSu4VgEnWvf28KqyHq8fEA3ew9u+vRZhcV/2X
SwJX0rE5626UDsieIVGP
=rgNN
-----END PGP SIGNATURE-----
Merge tag 'davinci-fixes-for-v4.13' of git://git.kernel.org/pub/scm/linux/kernel/git/nsekhar/linux-davinci into fixes
Pull "DaVinci fixes for v4.13" from Sekhar Nori:
Drop unused VPIF endpoints from device-tree.
They should be used only when an actual
remote-endpoint is connected.
* tag 'davinci-fixes-for-v4.13' of git://git.kernel.org/pub/scm/linux/kernel/git/nsekhar/linux-davinci:
ARM: dts: da850-lcdk: drop unused VPIF endpoints
ARM: dts: da850-evm: drop unused VPIF endpoints
Two fixes to correct the EMAC blocks memory region size to match the
datasheet. One that converts raw A83T clock indices to macros from the
clk dt-binding header, completing the A83T sunxi-ng clk driver.
-----BEGIN PGP SIGNATURE-----
iQJCBAABCgAsFiEE2nN1m/hhnkhOWjtHOJpUIZwPJDAFAll566wOHHdlbnNAY3Np
ZS5vcmcACgkQOJpUIZwPJDBBhw//RK8UpKhviq1kdn6yDNKe3dVYxRTGtISvpjJL
oPXpstt1b28DdbVjeIzh6jqF2HjjNmfFlLphOoOj1H7qKqhobr9+Cll05BNWbCmH
CIL6qL8FT7QyK8yw5sPLcdlNLkNil+KfgnrBf76Crdw0R1HXGENsw+gVE2lVX+tJ
yRqhguaQjWNLA5v+nfK4la+Ip1ddgKgiBmkpkq9tuR8hUDe33LHFZxur28EChHb/
P1HXL1DHUn2uO/tyk1TDhpYkNbOGiER4neOeQt21uaKLDaRMmkRlaNHgGg4CQ/fR
OYs24Sm0XB9oFVKzz2GttkIkdWTOqRI99VlXHEd3bU3Rc2PdClrdky615qkDn/90
CCjz7ibhnkri5+1/MWVC1iF/sotYnlCKjptXMZNRmqoyC2ol+tkNy84MoHvuWBCU
/IzCNuU8sGqC5/PjGNo2LwQNmzkFsk78s0MQAUyxG0isJwMrGQaSXGtOpSLJzUPv
MRBqYyK2CQk7gZtDyeZPepUUeQta/bAFIUROeozgTstvib86ywYrflrjkBTEyPar
Q+ptcoEgw1FVZZ4cQCGQWkg16dtFPSBMbGAbjlh/9DJXbZU/DVF5XRpTj8FthWm7
Vw/QQx4A18O+oS5/CkBjdltAb4RZIcF2ijAupRbwPIG+pzNVJVJ7JN7BzdVAFAyF
cf/Btkw=
=aFvT
-----END PGP SIGNATURE-----
Merge tag 'sunxi-fixes-for-4.13' of https://git.kernel.org/pub/scm/linux/kernel/git/sunxi/linux into fixes
Pull "Allwinner fixes for 4.13" from Chen-Yu Tsai:
Two fixes to correct the EMAC blocks memory region size to match the
datasheet. One that converts raw A83T clock indices to macros from the
clk dt-binding header, completing the A83T sunxi-ng clk driver.
* tag 'sunxi-fixes-for-4.13' of https://git.kernel.org/pub/scm/linux/kernel/git/sunxi/linux:
ARM: dts: sun8i: a83t: Switch to CCU device tree binding macros
arm64: allwinner: sun50i-a64: Correct emac register size
ARM: dts: sunxi: h3/h5: Correct emac register size
When resuming, set up registers that have been lost in the sleep state.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Fix deadlock in regulator quirk for R-Car Gen 2 SoCs
The da9063/da9210 regulator quirk for R-Car Gen2 boards uses a bus
notifier, and unregisters the notifier when it is no longer needed.
However, a notifier must not be unregistered from within the call chain.
This bug went unnoticed, as blocking_notifier_chain_unregister() didn't
take the semaphore during early boot. This is no longer the case as of
upstream commit 1c3c5eab17 ("sched/core: Enable might_sleep() and
smp_processor_id() checks early") and a deadlock occurs.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAABAgAGBQJZfw75AAoJENfPZGlqN0++i24P/j38Z5LKt529CBT2uPgdiIr7
lWEzU3Dgij/vPBUevz1s+hPa2i4jH1sfeKjSd2Ild3Ac+qAy6AwnzWlVYBCaBhlJ
qFAsr3nZ6ybftdA4rIJb1pWF+ugG7uNcQ3S0J5vLeiRU5HC/ZR+Ar+hjeuDwwIGa
889tI5UpKDFxYzkLIDVyiW0gX7WFhmxML0jptuV6c7bSl4Td/RxaP25Hcq8MsApv
oVre7mWPBKrIrjH26lPIUmdN4SGIHgUS3k2j9fFoWit9p2gYX6WDVbHjgALzPL1r
2Su8mSuP6ijutrrOuoB3OaNRlTaSr1Jpzfb6m6xRY+y1Anks0FxRVd712HrxinZK
4xW0ebUpfxpFsg8lJGoe1d1FGOSuj6A3/5GCrA+fQS0LSKthft+ovHmKBAogbSm3
7uCCU/h05LIkW0kDZYNjUqPWi+FwHZ4nEZEJiXMBt5DOyWpF5oIjCFeHmyp8VruI
xHZnq0pcck0Y+OSln2tiV43kw4QzK3BjTZZRc/IkVf19m24y+WJu9KfXTQhCDeLt
g4/zjAZrZddVQPtUz0f1YC07gPTmxLQpQJqLZBNeaed2W8p+tmUWUuoILpjPjEqR
Uu0EQE+t205YaxEz/ctYATdrpNOM5TWRhD+U25VTfhgxar3p9aC4GNVskL/fMGUy
/wTWYp1KVlxYjaxi3aOP
=N5ig
-----END PGP SIGNATURE-----
Merge tag 'renesas-fixes3-for-v4.13' of https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas into fixes
Pull "Third Round of Renesas ARM Based SoC Fixes for v4.13" from Simon Horman:
Fix deadlock in regulator quirk for R-Car Gen 2 SoCs
The da9063/da9210 regulator quirk for R-Car Gen2 boards uses a bus
notifier, and unregisters the notifier when it is no longer needed.
However, a notifier must not be unregistered from within the call chain.
This bug went unnoticed, as blocking_notifier_chain_unregister() didn't
take the semaphore during early boot. This is no longer the case as of
upstream commit 1c3c5eab17 ("sched/core: Enable might_sleep() and
smp_processor_id() checks early") and a deadlock occurs.
* tag 'renesas-fixes3-for-v4.13' of https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas:
ARM: shmobile: rcar-gen2: Fix deadlock in regulator quirk
All the fixes are for ARM64 mvebu:
- Fix the RTC interrupt on A7K/A8K which was missed when switching
from GIC to ICU
- Mark the A7K/A8K crypto engine as dma coherent
- Fix the number of GPIO on south bridge on Armada 3700
-----BEGIN PGP SIGNATURE-----
iIEEABECAEEWIQQYqXDMF3cvSLY+g9cLBhiOFHI71QUCWYHbmCMcZ3JlZ29yeS5j
bGVtZW50QGZyZWUtZWxlY3Ryb25zLmNvbQAKCRALBhiOFHI71ZhfAJ9Z9aMBG1xd
vbRYBDdqPdntruTudwCcDtM9jS2nL+207JfeMD0WMs7sDdU=
=XwFM
-----END PGP SIGNATURE-----
Merge tag 'mvebu-fixes-4.13-2' of git://git.infradead.org/linux-mvebu into fixes
Pull "mvebu fixes for 4.13 (part 2)" from Gregory CLEMENT:
All the fixes are for ARM64 mvebu:
- Fix the RTC interrupt on A7K/A8K which was missed when switching
from GIC to ICU
- Mark the A7K/A8K crypto engine as dma coherent
- Fix the number of GPIO on south bridge on Armada 3700
* tag 'mvebu-fixes-4.13-2' of git://git.infradead.org/linux-mvebu:
ARM64: dts: marvell: armada-37xx: Fix the number of GPIO on south bridge
arm64: dts: marvell: mark the cp110 crypto engine as dma coherent
arm64: dts: marvell: use ICU for the CP110 slave RTC
- 2 minor DT fixes
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAABCAAGBQJZf0PMAAoJEFk3GJrT+8ZlJskP/0JXcJt1I4myJv9A9z/+t0nh
7vT8H/ZwSyvK+/Iq1ccrf0Abe/c3445yHCwnF0T87R14A+oVVML5+uDolHIRZjlf
Wka34OoZFR9qTjphsFfQdZ0YcId7+a3BXNyvdkIWytWB+tzy/gpvlExBd54wJy0c
rceTEJsdmyObBFTmMTdlTzKQYLvurWoZ7d/N+T81UD3g0Xmq0EIOk0pEHSThK4SS
yQQWfRu7PCWDfXVu7qImalniaSa6wriHFC8hnrmaLfm2a9axfrTpvVknYuZUROlj
3T0Hgq3YHYe+BpS1P1Wryh60Yc8umbGy5JYg4GFOlNEGT9lBflk7nyG25aAbtwXY
BUU/r1cA1K+wR3XbUK9dpxW0GeCDTsQkHY9SA6+Aj5utNcMVhlnmGAHxJX9j1a8n
OhZtk339oCuBaf2Eg67OLvNGCPN21LAr2PL22LEGWkKSEUI61/1ZP1jjuCjN8Yk0
spXdFshdkYJyMqA4VuR05R5sBgqO25hXVDB2RYE4OuKINvAOfckgY9XpUhndxuYx
VVfX1up4bwoAUzVR+7XIsVrHaDR8pNS/Uv6kbhfB3X2c/jyPD1/Rc3T8p5DYirt7
DUPv8MJm/cwEC6uinQyqHTgB0tYLTWEY2+sv8NI2u8geYoJZO+Kt/12f3gcx296e
82LIlHnYKfYqUaqFwRRJ
=Z5d3
-----END PGP SIGNATURE-----
Merge tag 'amlogic-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/khilman/linux-amlogic into fixes
Pull "Amlogic fixes for v4.13-rc" from Kevin Hilman:
- 2 minor DT fixes
* tag 'amlogic-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/khilman/linux-amlogic:
ARM64: dts: meson-gxl-s905x-libretech-cc: fixup board definition
ARM64: dts: meson-gx: use specific compatible for the AO pwms
showed a wrong value, so fix it before it gets copy-pasted
to much.
-----BEGIN PGP SIGNATURE-----
iQFEBAABCAAuFiEE7v+35S2Q1vLNA3Lx86Z5yZzRHYEFAlmEM5wQHGhlaWtvQHNu
dGVjaC5kZQAKCRDzpnnJnNEdgVn+B/9Xy9km3t8cFVQ//SjNm1MEfiyVrgCnH6TH
trLit6AKq19LTIfeyI2RQucEXB0Csw2+YNh4q9Td6utnc0S3OpzueFsfKfAD9xif
WpRR6MJjCSlIQfhd09XuKyWoDwwNmnidv+Xd/H3M83dTOzoPmy0aDkn/shc1I+Tw
gL0CbLPz1HuHuG55S5EkxcBNi/Mj34qI5e3O8hrqzpfzmc1CdQ38zblfkiyu94v7
f/sVUv8p2YHyfvIJ1n1DxQx7lvAxGzo3Q7fXqS9D1jJx4iDWWFaraLMPu3NztLcv
1XrawJTdGZ9vTJSmzcEwH7JeAUHX41wIWjf/jEx9iE2R9EY0lyLo
=B0nU
-----END PGP SIGNATURE-----
Merge tag 'v4.13-rockchip-dts32fixes-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip into fixes
Pull "Rockchip dts32 fixes for 4.13" from Heiko Stübner:
Fix for the recently added mali dt support. The example
showed a wrong value, so fix it before it gets copy-pasted
to much.
* tag 'v4.13-rockchip-dts32fixes-1' of git://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip:
ARM: dts: rockchip: fix mali gpu node on rk3288
dt-bindings: gpu: drop wrong compatible from midgard binding example
Willem de Bruijn says:
====================
socket sendmsg MSG_ZEROCOPY
Introduce zerocopy socket send flag MSG_ZEROCOPY. This extends the
shared page support (SKBTX_SHARED_FRAG) from sendpage to sendmsg.
Implement the feature for TCP initially, as large writes benefit
most.
On a send call with MSG_ZEROCOPY, the kernel pins user pages and
links these directly into the skbuff frags[] array.
Each send call with MSG_ZEROCOPY that transmits data will eventually
queue a completion notification on the error queue: a per-socket u32
incremented on each such call. A request may have to revert to copy
to succeed, for instance when a device cannot support scatter-gather
IO. In that case a flag is passed along to notify that the operation
succeeded without zerocopy optimization.
The implementation extends the existing zerocopy infra for tuntap,
vhost and xen with features needed for TCP, notably reference
counting to handle cloning on retransmit and GSO.
For more details, see also the netdev 2.1 paper and presentation at
https://netdevconf.org/2.1/session.html?debruijn
Changelog:
v3 -> v4:
- dropped UDP, RAW and PF_PACKET for now
Without loopback support, datagrams are usually smaller than
the ~8KB size threshold needed to benefit from zerocopy.
- style: a few reverse chrismas tree
- minor: SO_ZEROCOPY returns ENOTSUPP on unsupported protocols
- minor: squashed SO_EE_CODE_ZEROCOPY_COPIED patch
- minor: rebased on top of net-next with kmap_atomic fix
v2 -> v3:
- fix rebase conflict: SO_ZEROCOPY 59 -> 60
v1 -> v2:
- fix (kbuild-bot): do not remove uarg until patch 5
- fix (kbuild-bot): move zerocopy_sg_from_iter doc with function
- fix: remove unused extern in header file
RFCv2 -> v1:
- patch 2
- review comment: in skb_copy_ubufs, always allocate order-0
page, also when replacing compound source pages.
- patch 3
- fix: always queue completion notification on MSG_ZEROCOPY,
also if revert to copy.
- fix: on syscall abort, correctly revert notification state
- minor: skip queue notification on SOCK_DEAD
- minor: replace BUG_ON with WARN_ON in recoverable error
- patch 4
- new: add socket option SOCK_ZEROCOPY.
only honor MSG_ZEROCOPY if set, ignore for legacy apps.
- patch 5
- fix: clear zerocopy state on skb_linearize
- patch 6
- fix: only coalesce if prev errqueue elem is zerocopy
- minor: try coalescing with list tail instead of head
- minor: merge bytelen limit patch
- patch 7
- new: signal when data had to be copied
- patch 8 (tcp)
- optimize: avoid setting PSH bit when exceeding max frags.
that limits GRO on the client. do not goto new_segment.
- fix: fail on MSG_ZEROCOPY | MSG_FASTOPEN
- minor: do not wait for memory: does not work for optmem
- minor: simplify alloc
- patch 9 (udp)
- new: add PF_INET6
- fix: attach zerocopy notification even if revert to copy
- minor: simplify alloc size arithmetic
- patch 10 (raw hdrinc)
- new: add PF_INET6
- patch 11 (pf_packet)
- minor: simplify slightly
- patch 12
- new msg_zerocopy regression test: use veth pair to test
all protocols: ipv4/ipv6/packet, tcp/udp/raw, cork
all relevant ethtool settings: rx off, sg off
all relevant packet lengths: 0, <MAX_HEADER, max size
RFC -> RFCv2:
- review comment: do not loop skb with zerocopy frags onto rx:
add skb_orphan_frags_rx to orphan even refcounted frags
call this in __netif_receive_skb_core, deliver_skb and tun:
same as commit 1080e512d4 ("net: orphan frags on receive")
- fix: hold an explicit sk reference on each notification skb.
previously relied on the reference (or wmem) held by the
data skb that would trigger notification, but this breaks
on skb_orphan.
- fix: when aborting a send, do not inc the zerocopy counter
this caused gaps in the notification chain
- fix: in packet with SOCK_DGRAM, pull ll headers before calling
zerocopy_sg_from_iter
- fix: if sock_zerocopy_realloc does not allow coalescing,
do not fail, just allocate a new ubuf
- fix: in tcp, check return value of second allocation attempt
- chg: allocate notification skbs from optmem
to avoid affecting tcp write queue accounting (TSQ)
- chg: limit #locked pages (ulimit) per user instead of per process
- chg: grow notification ids from 16 to 32 bit
- pass range [lo, hi] through 32 bit fields ee_info and ee_data
- chg: rebased to davem-net-next on top of v4.10-rc7
- add: limit notification coalescing
sharing ubufs limits overhead, but delays notification until
the last packet is released, possibly unbounded. Add a cap.
- tests: add snd_zerocopy_lo pf_packet test
- tests: two bugfixes (add do_flush_tcp, ++sent not only in debug)
Limitations / Known Issues:
- TCP may build slightly smaller than max TSO packets due to
exceeding MAX_SKB_FRAGS frags when zerocopy pages are unaligned.
- All SKBTX_SHARED_FRAG may require additional __skb_linearize or
skb_copy_ubufs calls in u32, skb_find_text, similar to
skb_checksum_help.
Notification skbuffs are allocated from optmem. For sockets that
cannot effectively coalesce notifications, the optmem max may need
to be increased to avoid hitting -ENOBUFS:
sysctl -w net.core.optmem_max=1048576
In application load, copy avoidance shows a roughly 5% systemwide
reduction in cycles when streaming large flows and a 4-8% reduction in
wall clock time on early tensorflow test workloads.
For the single-machine veth tests to succeed, loopback support has to
be temporarily enabled by making skb_orphan_frags_rx map to
skb_orphan_frags.
* Performance
The below table shows cycles reported by perf for a netperf process
sending a single 10 Gbps TCP_STREAM. The first three columns show
Mcycles spent in the netperf process context. The second three columns
show time spent systemwide (-a -C A,B) on the two cpus that run the
process and interrupt handler. Reported is the median of at least 3
runs. std is a standard netperf, zc uses zerocopy and % is the ratio.
Netperf is pinned to cpu 2, network interrupts to cpu3, rps and rfs
are disabled and the kernel is booted with idle=halt.
NETPERF=./netperf -t TCP_STREAM -H $host -T 2 -l 30 -- -m $size
perf stat -e cycles $NETPERF
perf stat -C 2,3 -a -e cycles $NETPERF
--process cycles-- ----cpu cycles----
std zc % std zc %
4K 27,609 11,217 41 49,217 39,175 79
16K 21,370 3,823 18 43,540 29,213 67
64K 20,557 2,312 11 42,189 26,910 64
256K 21,110 2,134 10 43,006 27,104 63
1M 20,987 1,610 8 42,759 25,931 61
Perf record indicates the main source of these differences. Process
cycles only at 1M writes (perf record; perf report -n):
std:
Samples: 42K of event 'cycles', Event count (approx.): 21258597313
79.41% 33884 netperf [kernel.kallsyms] [k] copy_user_generic_string
3.27% 1396 netperf [kernel.kallsyms] [k] tcp_sendmsg
1.66% 694 netperf [kernel.kallsyms] [k] get_page_from_freelist
0.79% 325 netperf [kernel.kallsyms] [k] tcp_ack
0.43% 188 netperf [kernel.kallsyms] [k] __alloc_skb
zc:
Samples: 1K of event 'cycles', Event count (approx.): 1439509124
30.36% 584 netperf.zerocop [kernel.kallsyms] [k] gup_pte_range
14.63% 284 netperf.zerocop [kernel.kallsyms] [k] __zerocopy_sg_from_iter
8.03% 159 netperf.zerocop [kernel.kallsyms] [k] skb_zerocopy_add_frags_iter
4.84% 96 netperf.zerocop [kernel.kallsyms] [k] __alloc_skb
3.10% 60 netperf.zerocop [kernel.kallsyms] [k] kmem_cache_alloc_node
* Safety
The number of pages that can be pinned on behalf of a user with
MSG_ZEROCOPY is bound by the locked memory ulimit.
While the kernel holds process memory pinned, a process cannot safely
reuse those pages for other purposes. Packets looped onto the receive
stack and queued to a socket can be held indefinitely. Avoid unbounded
notification latency by restricting user pages to egress paths only.
skb_orphan_frags_rx() will create a private copy of pages even for
refcounted packets when these are looped, as did skb_orphan_frags for
the original tun zerocopy implementation.
Pages are not remapped read-only. Processes can modify packet contents
while packets are in flight in the kernel path. Bytes on which kernel
control flow depends (headers) are copied to avoid TOCTTOU attacks.
Datapath integrity does not otherwise depend on payload, with three
exceptions: checksums, optional sk_filter/tc u32/.. and device +
driver logic. The effect of wrong checksums is limited to the
misbehaving process. TC filters that access contents may have to be
excluded by adding an skb_orphan_frags_rx.
Processes can also safely avoid OOM conditions by bounding the number
of bytes passed with MSG_ZEROCOPY and by removing shared pages after
transmission from their own memory map.
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
Introduce regression test for msg_zerocopy feature. Send traffic from
one process to another with and without zerocopy.
Evaluate tcp, udp, raw and packet sockets, including variants
- udp: corking and corking with mixed copy/zerocopy calls
- raw: with and without hdrincl
- packet: at both raw and dgram level
Test on both ipv4 and ipv6, optionally with ethtool changes to
disable scatter-gather, tx checksum or tso offload. All of these
can affect zerocopy behavior.
The regression test can be run on a single machine if over a veth
pair. Then skb_orphan_frags_rx must be modified to be identical to
skb_orphan_frags to allow forwarding zerocopy locally.
The msg_zerocopy.sh script will setup the veth pair in network
namespaces and run all tests.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Enable support for MSG_ZEROCOPY to the TCP stack. TSO and GSO are
both supported. Only data sent to remote destinations is sent without
copying. Packets looped onto a local destination have their payload
copied to avoid unbounded latency.
Tested:
A 10x TCP_STREAM between two hosts showed a reduction in netserver
process cycles by up to 70%, depending on packet size. Systemwide,
savings are of course much less pronounced, at up to 20% best case.
msg_zerocopy.sh 4 tcp:
without zerocopy
tx=121792 (7600 MB) txc=0 zc=n
rx=60458 (7600 MB)
with zerocopy
tx=286257 (17863 MB) txc=286257 zc=y
rx=140022 (17863 MB)
This test opens a pair of sockets over veth, one one calls send with
64KB and optionally MSG_ZEROCOPY and on the other reads the initial
bytes. The receiver truncates, so this is strictly an upper bound on
what is achievable. It is more representative of sending data out of
a physical NIC (when payload is not touched, either).
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Bound the number of pages that a user may pin.
Follow the lead of perf tools to maintain a per-user bound on memory
locked pages commit 789f90fcf6 ("perf_counter: per user mlock gift")
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
In the simple case, each sendmsg() call generates data and eventually
a zerocopy ready notification N, where N indicates the Nth successful
invocation of sendmsg() with the MSG_ZEROCOPY flag on this socket.
TCP and corked sockets can cause send() calls to append new data to an
existing sk_buff and, thus, ubuf_info. In that case the notification
must hold a range. odify ubuf_info to store a inclusive range [N..N+m]
and add skb_zerocopy_realloc() to optionally extend an existing range.
Also coalesce notifications in this common case: if a notification
[1, 1] is about to be queued while [0, 0] is the queue tail, just modify
the head of the queue to read [0, 1].
Coalescing is limited to a few TSO frames worth of data to bound
notification latency.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The send call ignores unknown flags. Legacy applications may already
unwittingly pass MSG_ZEROCOPY. Continue to ignore this flag unless a
socket opts in to zerocopy.
Introduce socket option SO_ZEROCOPY to enable MSG_ZEROCOPY processing.
Processes can also query this socket option to detect kernel support
for the feature. Older kernels will return ENOPROTOOPT.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The kernel supports zerocopy sendmsg in virtio and tap. Expand the
infrastructure to support other socket types. Introduce a completion
notification channel over the socket error queue. Notifications are
returned with ee_origin SO_EE_ORIGIN_ZEROCOPY. ee_errno is 0 to avoid
blocking the send/recv path on receiving notifications.
Add reference counting, to support the skb split, merge, resize and
clone operations possible with SOCK_STREAM and other socket types.
The patch does not yet modify any datapaths.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Refine skb_copy_ubufs to support compound pages. With upcoming TCP
zerocopy sendmsg, such fragments may appear.
The existing code replaces each page one for one. Splitting each
compound page into an independent number of regular pages can result
in exceeding limit MAX_SKB_FRAGS if data is not exactly page aligned.
Instead, fill all destination pages but the last to PAGE_SIZE.
Split the existing alloc + copy loop into separate stages:
1. compute bytelength and minimum number of pages to store this.
2. allocate
3. copy, filling each page except the last to PAGE_SIZE bytes
4. update skb frag array
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Add sock_omalloc and sock_ofree to be able to allocate control skbs,
for instance for looping errors onto sk_error_queue.
The transmit budget (sk_wmem_alloc) is involved in transmit skb
shaping, most notably in TCP Small Queues. Using this budget for
control packets would impact transmission.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
If the decrementer wraps again and de-asserts the decrementer
exception while hard-disabled, __check_irq_replay() has a test to
notice the wrap when interrupts are re-enabled.
The decrementer check must be done when clearing the PACA_IRQ_HARD_DIS
flag, not when the PACA_IRQ_DEC flag is tested. Previously this worked
because the decrementer interrupt was always the first one checked
after clearing the hard disable flag, but HMI check was moved ahead of
that, which introduced this bug.
This can cause a missed decrementer interrupt if we soft-disable
interrupts then take an HMI which is recorded in irq_happened, then
hard-disable interrupts for > 4s to wrap the decrementer.
Fixes: e0e0d6b739 ("powerpc/64: Replay hypervisor maintenance interrupt first")
Cc: stable@vger.kernel.org # v4.9+
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
POWER9 DD2 PMU can stop after a state-loss idle in some conditions.
A solution is to set then clear MMCRA[60] after wake from state-loss
idle. MMCRA[60] is a non-architected bit, see the user manual for
details.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Acked-by: Madhavan Srinivasan <maddy@linux.vnet.ibm.com>
Reviewed-by: Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>
Acked-by: Anton Blanchard <anton@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Just a few small fixes for 4.13.
* 'drm-fixes-4.13' of git://people.freedesktop.org/~agd5f/linux:
drm/amdgpu: Use list_del_init in amdgpu_mn_unregister
drm/amdgpu: Fix undue fallthroughs in golden registers initialization
drm/amdgpu: fix header on gfx9 clear state
Jiri Pirko says:
====================
mlxsw: Support for IPv6 UC router
Ido says:
This set adds support for IPv6 unicast routes offload. The first four
patches make the FIB notification chain generic so that it could be used
by address families other than IPv4. This is done by having each address
family register its callbacks with the common code, so that its FIB tables
and rules could be dumped upon registration to the chain, while ensuring
the integrity of the dump. The exact mechanics are explained in detail in
the first patch.
The next six patches build upon this work and add the necessary callbacks
in IPv6 code. This allows listeners of the chain to receive notifications
about IPv6 routes addition, deletion and replacement as well as FIB rules
notifications.
Unlike user space notifications for IPv6 multipath routes, the FIB
notification chain notifies these on a per-nexthop basis. This allows
us to keep the common code lean and is also unnecessary, as notifications
are serialized by each table's lock whereas applications maintaining
netlink caches may suffer from concurrent dumps and deletions / additions
of routes.
The next five patches audit the different code paths reading the route's
reference count (rt6i_ref) and remove assumptions regarding its meaning.
This is needed since non-FIB users need to be able to hold a reference on
the route and a non-zero reference count no longer means the route is in
the FIB.
The last six patches enable the mlxsw driver to offload IPv6 unicast
routes to the Spectrum ASIC. Without resorting to ACLs, lookup is done
solely based on the destination IP, so the abort mechanism is invoked
upon the addition of source-specific routes.
Follow-up patch sets will increase the scale of gatewayed routes by
consolidating identical nexthop groups to one adjacency entry in the
device's adjacency table (as in IPv4), as well as add support for
NH_{ADD,DEL} events which enable support for the
'ignore_routes_with_linkdown' sysctl.
Changes in v2:
* Provide offload indication for individual nexthops (David Ahern).
* Use existing route reference count instead of adding another one.
This resulted in several new patches to remove assumptions regarding
current semantics of the existing reference count (David Ahern).
* Add helpers to allow non-FIB users to take a reference on route.
* Remove use of tb6_lock in mlxsw (David Ahern).
* Add IPv6 dependency to mlxsw.
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
We now have all the necessary IPv6 infrastructure in place, so stop
ignoring these notifications.
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Without resorting to ACLs, the device performs route lookup solely based
on the destination IP address.
In case source-specific routing is needed, an error is returned and the
abort mechanism is activated, thus allowing the kernel to take over
forwarding decisions.
Instead of aborting, we can trap specific destination prefixes where
source-specific routes are present, but this will result in a lot more
code that is unlikely to ever be used.
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
In case we got a replace event, then the replaced route must exist. If
the route isn't capable of multipath, then replace first matching
non-multipath capable route.
If the route is capable of multipath and matching multipath capable
route is found, then replace it. Otherwise, replace first matching
non-multipath capable route.
The new route is inserted before the replaced one. In case the replaced
route is currently offloaded, then it's overwritten in the device's table
by the new route and later deleted, thus not impacting routed traffic.
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Allow directly connected and remote unicast IPv6 routes to be programmed
to the device's tables.
As with IPv4, identical routes - sharing the same destination prefix -
are ordered in a FIB node according to their table ID and then the
metric. While the kernel doesn't share the same trie for the local and
main table, this does happen in the device, so ordering according to
table ID is needed.
Since individual nexthops can be added and deleted in IPv6, each FIB
entry stores a linked list of the rt6_info structs it represents. Upon
the addition or deletion of a nexthop, a new nexthop group is allocated
according to the new configuration and the old one is destroyed.
Identical groups aren't currently consolidated, but will be in a
follow-up patchset.
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
We only allow FIB offload in the presence of default rules or an l3mdev
rule. In a similar fashion to IPv4 FIB rules, sanitize IPv6 rules.
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
