When trying to rescan disks in petitboot shell, we hit the following
softlockup stacktrace:
Kernel panic - not syncing: System is deadlocked on memory
[ 241.223394] CPU: 32 PID: 693 Comm: sh Not tainted 5.4.16-openpower1 #1
[ 241.223406] Call Trace:
[ 241.223415] [c0000003f07c3180] [c000000000493fc4] dump_stack+0xa4/0xd8 (unreliable)
[ 241.223432] [c0000003f07c31c0] [c00000000007d4ac] panic+0x148/0x3cc
[ 241.223446] [c0000003f07c3260] [c000000000114b10] out_of_memory+0x468/0x4c4
[ 241.223461] [c0000003f07c3300] [c0000000001472b0] __alloc_pages_slowpath+0x594/0x6d8
[ 241.223476] [c0000003f07c3420] [c00000000014757c] __alloc_pages_nodemask+0x188/0x1a4
[ 241.223492] [c0000003f07c34a0] [c000000000153e10] alloc_pages_current+0xcc/0xd8
[ 241.223508] [c0000003f07c34e0] [c0000000001577ac] alloc_slab_page+0x30/0x98
[ 241.223524] [c0000003f07c3520] [c0000000001597fc] new_slab+0x138/0x40c
[ 241.223538] [c0000003f07c35f0] [c00000000015b204] ___slab_alloc+0x1e4/0x404
[ 241.223552] [c0000003f07c36c0] [c00000000015b450] __slab_alloc+0x2c/0x48
[ 241.223566] [c0000003f07c36f0] [c00000000015b754] kmem_cache_alloc_node+0x9c/0x1b4
[ 241.223582] [c0000003f07c3760] [c000000000218c48] blk_alloc_queue_node+0x34/0x270
[ 241.223599] [c0000003f07c37b0] [c000000000226574] blk_mq_init_queue+0x2c/0x78
[ 241.223615] [c0000003f07c37e0] [c0000000002ff710] scsi_mq_alloc_queue+0x28/0x70
[ 241.223631] [c0000003f07c3810] [c0000000003005b8] scsi_alloc_sdev+0x184/0x264
[ 241.223647] [c0000003f07c38a0] [c000000000300ba0] scsi_probe_and_add_lun+0x288/0xa3c
[ 241.223663] [c0000003f07c3a00] [c000000000301768] __scsi_scan_target+0xcc/0x478
[ 241.223679] [c0000003f07c3b20] [c000000000301c64] scsi_scan_channel.part.9+0x74/0x7c
[ 241.223696] [c0000003f07c3b70] [c000000000301df4] scsi_scan_host_selected+0xe0/0x158
[ 241.223712] [c0000003f07c3bd0] [c000000000303f04] store_scan+0x104/0x114
[ 241.223727] [c0000003f07c3cb0] [c0000000002d5ac4] dev_attr_store+0x30/0x4c
[ 241.223741] [c0000003f07c3cd0] [c0000000001dbc34] sysfs_kf_write+0x64/0x78
[ 241.223756] [c0000003f07c3cf0] [c0000000001da858] kernfs_fop_write+0x170/0x1b8
[ 241.223773] [c0000003f07c3d40] [c0000000001621fc] __vfs_write+0x34/0x60
[ 241.223787] [c0000003f07c3d60] [c000000000163c2c] vfs_write+0xa8/0xcc
[ 241.223802] [c0000003f07c3db0] [c000000000163df4] ksys_write+0x70/0xbc
[ 241.223816] [c0000003f07c3e20] [c00000000000b40c] system_call+0x5c/0x68
As a part of the scan process Linux will allocate and configure a
scsi_device for each target to be scanned. If the device is not present,
then the scsi_device is torn down. As a part of scsi_device teardown a
workqueue item will be scheduled and the lockups we see are because there
are 250k workqueue items to be processed. Accoding to the specification of
SIS-64 sas controller, max_channel should be decreased on SIS-64 adapters
to 4.
The patch fixes softlockup issue.
Thanks for Oliver Halloran's help with debugging and explanation!
Link: https://lore.kernel.org/r/1583510248-23672-1-git-send-email-wenxiong@linux.vnet.ibm.com
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Based on 1 normalized pattern(s):
this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license as published by
the free software foundation either version 2 of the license or at
your option any later version this program is distributed in the
hope that it will be useful but without any warranty without even
the implied warranty of merchantability or fitness for a particular
purpose see the gnu general public license for more details you
should have received a copy of the gnu general public license along
with this program if not write to the free software foundation inc
59 temple place suite 330 boston ma 02111 1307 usa
extracted by the scancode license scanner the SPDX license identifier
GPL-2.0-or-later
has been chosen to replace the boilerplate/reference in 1334 file(s).
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Allison Randal <allison@lohutok.net>
Reviewed-by: Richard Fontana <rfontana@redhat.com>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190527070033.113240726@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
While dlpar adding primary ipr adapter back, driver goes through adapter
initialization then schedule ipr_worker_thread to start te disk scan by
dropping the host lock, calling scsi_add_device. Then get the adapter reset
request again, so driver does scsi_block_requests, this will cause the
scsi_add_device get hung until we unblock. But we can't run ipr_worker_thread
to do the unblock because its stuck in scsi_add_device.
This patch fixes the issue.
[mkp: typo and whitespace fixes]
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This patch adds formatting error overlay 0x41 to improve debug
capabilities.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Use the sgl_alloc_order() and sgl_free_order() functions instead of open
coding these functions.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Bump driver version
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wendy Xiong <wenxiong@linux.vnet.ibm.com>
Tested-by: Wendy Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Switch the ipr driver to use pci_alloc_irq_vectors. We need to two
calls to pci_alloc_irq_vectors as ipr only supports multiple MSI-X
vectors, but not multiple MSI vectors.
Otherwise this cleans up a lot of cruft and allows to use a common
request_irq loop for irq types, which happens to only iterate over a
single line in the non MSI-X case.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
A 9084 error gets logged by the ipr adapter when adapter raw mode gets
enabled. A bunch of unformatted hex data also gets logged for this
error, which is of little use, so let's avoid logging it by default in
order to avoid the log getting polluted with useless data.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This patch implements functions for pushing HCAM (host controlled
asynchronous messages) error buffers to userspace through sysfs
attributes. Reads to the "async_err_log" attribute will result in a
single HCAM buffer being copied to userspace; one can process the next
HCAM buffer by writing any string to the same attribute.
A new list was added to the ioa_cfg structure to store the HCAM buffers
for later reporting. We also send a KOBJ_CHANGE event whenever a new
HCAM buffer is made available to userspace.
Signed-off-by: Heitor Ricardo Alves de Siqueira <halves@linux.vnet.ibm.com>
Signed-off-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
When performing an async scan, make sure the kthread doing scanning
doesn't start before the scsi host is fully initialized.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Add the appropriate definitions and table entries for new adapters
support.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The new name is irq_poll as iopoll is already taken. Better suggestions
welcome.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com>
Signed-off-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Some new adapters require a special Configure Cache Parameters command
to enable the adapter write cache, so send this during the adapter
initialization if the adapter requires it.
Signed-off-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Add an IOA Inquiry command for Page 0xC4 during IOA initialization to
collect cache capabilities, particularly to check if Sync IOA Write
Cache is supported.
Inquiry will happen right after Cap Inquiry on page 0xD0; and will
execute only if the "Supported Pages" field in Inquiry Page 0x0 shows
support for Page 0xC4. Otherwise, assume Sync IOA Write Cache is
not supported.
Signed-off-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Add a holding pattern prior to collecting dump data, to wait for the IOA
indication that the Mailbox register is stable and won't change without
an explicit reset. This ensures we'll be collecting meaningful dump
data, even when dumping right after an adapter reset.
In the event of a timeout, we still force the dump, since a partial dump
still might be useful.
Signed-off-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Some misc fixes for endianness checking with sparse so sparse with
endian checking now runs clean. Fixes a minor bug in the process
which was uncovered by sparse which would result in unnecessary
error recovery for check conditions.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
When ipr's internal driver trace was changed to an atomic, a signed/unsigned
bug slipped in which results in us indexing backwards in our memory buffer
writing on memory that does not belong to us. This patch fixes this by removing
the modulo and instead just mask off the low bits.
Cc: <stable@vger.kernel.org>
Tested-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Reviewed-by: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
Increase the default init stage change timeout from 15 seconds to 30 seconds.
This resolves issues we have seen with some adapters not transitioning
to the first init stage within 15 seconds, which results in adapter
initialization failures.
Cc: <stable@vger.kernel.org>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
This patch implements raw mode support for AF DASD in ipr driver
which allows for tools to send commands directly to physical
devices which are members of RAID arrays when enabled in the firmware.
[jejb: fix up whitespace]
Signed-off-by: Wen Xiong<wenxiong@linux.vnet.ibm.com>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Daniel Kreling <kreling@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
The pci_set_pcie_reset_state has changed semantics to not be callable
from interrupt context, so change ipr's usage of the API to comply with
this change by ensuring this occurs from a workqueue.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Reviewed-by: Daniel Kreling <kreling@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
Currently when performing a reboot with an ipr adapter,
the adapter gets shutdown completely, flushing all write
cache, as well as performing a full hardware reset of the card
during the shutdown phase of the old kernel. This ensures
the adapter is in a fully quiesced state across the reboot.
There are scenarios, however, such as when performing
kexec, where this full adapter shutdown is not required
and not desired, since it can make the reboot process take
noticeably longer.
This patch adds a module parameter to allow for skipping the
full shutdown during reboot. Rather than performing a full
adapter shutdown and reset, we simply cancel any outstanding
error buffers, place the adapter into a state where it has no
memory of any DMA addresses from the old kernel, then disable
the device. This significantly speeds up kexec boot, particularly
in configurations with multiple ipr adapters.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Reviewed-by: Daniel Kreling <kreling@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
Fixes a race condition in abort handling that was injected
when multiple interrupt support was added. When only a single
interrupt is present, the adapter guarantees it will send
responses for aborted commands prior to the response for the
abort command itself. With multiple interrupts, these responses
generally come back on different interrupts, so we need to
ensure the abort thread waits until the aborted command is
complete so we don't perform a double completion. This race
condition was being hit frequently in environments which
were triggering command timeouts, which was resulting in
a double completion causing a kernel oops.
Cc: <stable@vger.kernel.org>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wendy Xiong <wenxiong@linux.vnet.ibm.com>
Tested-by: Wendy Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Switch device scanning logic in the ipr driver to use
the async scan API. This speeds up boot times, particularly
on large systems.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Reviewed-by: Wen Xiong<wenxiong@linux.vnet.ibm.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Even though the ipr driver is only used on PCI, convert it
to use the generic DMA API.
Signed-off-by: Anton Blanchard <anton@samba.org>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Fix a typo in the IPR_IOASC_HW_CMD_FAILED declaration. Based on a patch
from Wen Xiong <wenxiong@linux.vnet.ibm.com>.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Failing Device information are logged when IOA firmware detected these
illegal request such as IOA firmware doesn't support inquiry with page
code 2. The patch fixes the issue.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Tested-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Add the appropriate definition and table entry for new hardware support.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
If, when the ipr driver loads, the adapter is in an EEH error state,
it will currently oops and not be able to recover, as it attempts
to access memory that has not yet been allocated. We've seen this
occur in some kexec scenarios. The following patch fixes the oops
and also allows the driver to recover from these probe time EEH errors.
[jejb: checkpatch fix]
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Add the appropriate definition and table entry for new hardware support.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch removes extended delay bit on GSCSI reads/writes ops, the
performance will be significanly better.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
To help debugging several new SAS controllers, increase the dump size in ipr
driver to 80MB.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Add the appropriate definitions and table entries for new adapter support.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Defined target_ids,array_ids and vsets_ids as unsigned long to avoid
target_destroy accessing memory after it was freed.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Driver passed the wrong IOADL address to IOA adapter. The patch
fixes the issue.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
When resetting an ipr adapter, we use scsi_block_requests to
block any new commands from scsi core, and then unblock after
the reset. When hotplug removing an adapter, we shut it down
and go through this same code, but we've seen issues with
scsi_unblock_requests running after the adapter's memory has
been freed. There is really no need to block/unblock when
the adapter is being removed, so this patch skips the
block/unblock and will immediately fail any commands that
happen to make it to queuecommand while the adapter is
being shutdown.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch implements blk iopoll in ipr driver for performance improvement.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch reduces lock contention while implementing distributed
completion processing.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
The new generation IBM SAS Controllers will support MSI-X interrupts and
Distributed Completion Processing features. This patch add these support
in ipr device driver.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
The resource path as displayed by the ipr driver is the
location string identifying a location on the SAS fabric.
This patch adds the SCSI host number such that error logs
can be more easily correlated in multiple adapter configurations.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Add the appropriate definitions and table entries for new adapter support.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Acked-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Reduce the amount of time the host lock is held in the interrupt handler
for improved performance.
[jejb: fix up checkpatch noise]
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
