linux_dsm_epyc7002/drivers/gpu/drm/i915/intel_guc_submission.c

1344 lines
39 KiB
C
Raw Normal View History

/*
* Copyright © 2014 Intel Corporation
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice (including the next
* paragraph) shall be included in all copies or substantial portions of the
* Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*
*/
#include <linux/circ_buf.h>
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
#include <trace/events/dma_fence.h>
#include "intel_guc_submission.h"
#include "intel_lrc_reg.h"
#include "i915_drv.h"
#define GUC_PREEMPT_FINISHED 0x1
#define GUC_PREEMPT_BREADCRUMB_DWORDS 0x8
#define GUC_PREEMPT_BREADCRUMB_BYTES \
(sizeof(u32) * GUC_PREEMPT_BREADCRUMB_DWORDS)
/**
* DOC: GuC-based command submission
*
* GuC client:
* A intel_guc_client refers to a submission path through GuC. Currently, there
* are two clients. One of them (the execbuf_client) is charged with all
* submissions to the GuC, the other one (preempt_client) is responsible for
* preempting the execbuf_client. This struct is the owner of a doorbell, a
* process descriptor and a workqueue (all of them inside a single gem object
* that contains all required pages for these elements).
*
* GuC stage descriptor:
* During initialization, the driver allocates a static pool of 1024 such
* descriptors, and shares them with the GuC.
* Currently, there exists a 1:1 mapping between a intel_guc_client and a
* guc_stage_desc (via the client's stage_id), so effectively only one
* gets used. This stage descriptor lets the GuC know about the doorbell,
* workqueue and process descriptor. Theoretically, it also lets the GuC
* know about our HW contexts (context ID, etc...), but we actually
* employ a kind of submission where the GuC uses the LRCA sent via the work
* item instead (the single guc_stage_desc associated to execbuf client
* contains information about the default kernel context only, but this is
* essentially unused). This is called a "proxy" submission.
*
* The Scratch registers:
* There are 16 MMIO-based registers start from 0xC180. The kernel driver writes
* a value to the action register (SOFT_SCRATCH_0) along with any data. It then
* triggers an interrupt on the GuC via another register write (0xC4C8).
* Firmware writes a success/fail code back to the action register after
* processes the request. The kernel driver polls waiting for this update and
* then proceeds.
* See intel_guc_send()
*
* Doorbells:
* Doorbells are interrupts to uKernel. A doorbell is a single cache line (QW)
* mapped into process space.
*
* Work Items:
* There are several types of work items that the host may place into a
* workqueue, each with its own requirements and limitations. Currently only
* WQ_TYPE_INORDER is needed to support legacy submission via GuC, which
* represents in-order queue. The kernel driver packs ring tail pointer and an
* ELSP context descriptor dword into Work Item.
* See guc_add_request()
*
*/
static inline struct i915_priolist *to_priolist(struct rb_node *rb)
{
return rb_entry(rb, struct i915_priolist, node);
}
static inline bool is_high_priority(struct intel_guc_client *client)
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
{
return (client->priority == GUC_CLIENT_PRIORITY_KMD_HIGH ||
client->priority == GUC_CLIENT_PRIORITY_HIGH);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
}
static int reserve_doorbell(struct intel_guc_client *client)
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
{
unsigned long offset;
unsigned long end;
u16 id;
GEM_BUG_ON(client->doorbell_id != GUC_DOORBELL_INVALID);
/*
* The bitmap tracks which doorbell registers are currently in use.
* It is split into two halves; the first half is used for normal
* priority contexts, the second half for high-priority ones.
*/
offset = 0;
end = GUC_NUM_DOORBELLS / 2;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (is_high_priority(client)) {
offset = end;
end += offset;
}
id = find_next_zero_bit(client->guc->doorbell_bitmap, end, offset);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (id == end)
return -ENOSPC;
__set_bit(id, client->guc->doorbell_bitmap);
client->doorbell_id = id;
DRM_DEBUG_DRIVER("client %u (high prio=%s) reserved doorbell: %d\n",
client->stage_id, yesno(is_high_priority(client)),
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
id);
return 0;
}
static bool has_doorbell(struct intel_guc_client *client)
{
if (client->doorbell_id == GUC_DOORBELL_INVALID)
return false;
return test_bit(client->doorbell_id, client->guc->doorbell_bitmap);
}
static void unreserve_doorbell(struct intel_guc_client *client)
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
{
GEM_BUG_ON(!has_doorbell(client));
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
__clear_bit(client->doorbell_id, client->guc->doorbell_bitmap);
client->doorbell_id = GUC_DOORBELL_INVALID;
}
/*
* Tell the GuC to allocate or deallocate a specific doorbell
*/
static int __guc_allocate_doorbell(struct intel_guc *guc, u32 stage_id)
{
u32 action[] = {
INTEL_GUC_ACTION_ALLOCATE_DOORBELL,
stage_id
};
return intel_guc_send(guc, action, ARRAY_SIZE(action));
}
static int __guc_deallocate_doorbell(struct intel_guc *guc, u32 stage_id)
{
u32 action[] = {
INTEL_GUC_ACTION_DEALLOCATE_DOORBELL,
stage_id
};
return intel_guc_send(guc, action, ARRAY_SIZE(action));
}
static struct guc_stage_desc *__get_stage_desc(struct intel_guc_client *client)
{
struct guc_stage_desc *base = client->guc->stage_desc_pool_vaddr;
return &base[client->stage_id];
}
/*
* Initialise, update, or clear doorbell data shared with the GuC
*
* These functions modify shared data and so need access to the mapped
* client object which contains the page being used for the doorbell
*/
static void __update_doorbell_desc(struct intel_guc_client *client, u16 new_id)
{
struct guc_stage_desc *desc;
drm/i915/guc: refactor doorbell management code This patch refactors the driver's handling and tracking of doorbells, in preparation for a later one which will resolve a suspend-resume issue. There are three resources to be managed: 1. Cachelines: a single line within the client-object's page 0 is snooped by doorbell hardware for writes from the host. 2. Doorbell registers: each defines one cacheline to be snooped. 3. Bitmap: tracks which doorbell registers are in use. The doorbell setup/teardown protocol starts with: 1. Pick a cacheline: select_doorbell_cacheline() 2. Find an available doorbell register: assign_doorbell() (These values are passed to the GuC via the shared context descriptor; this part of the sequence remains unchanged). 3. Update the bitmap to reflect registers-in-use 4. Prepare the cacheline for use by setting its status to ENABLED 5. Ask the GuC to program the doorbell to snoop the cacheline and of course teardown is very similar: 6. Set the cacheline to DISABLED 7. Ask the GuC to reprogram the doorbell to stop snooping 8. Record that the doorbell is not in use. Operations 6-8 (guc_disable_doorbell(), host2guc_release_doorbell(), and release_doorbell()) were called in sequence from guc_client_free(), but are now moved into the teardown phase of the common function. Steps 4-5 (guc_init_doorbell() and host2guc_allocate_doorbell()) were similarly done as sequential steps in guc_client_alloc(), but since it turns out that we don't need to be able to do them separately they're now collected into the setup phase of the common function. The only new code (and new capability) is the block tagged /* Update the GuC's idea of the doorbell ID */ i.e. we can now *change* the doorbell register used by an existing client, whereas previously it was set once for the entire lifetime of the client. We will use this new feature in the next patch. v2: Trivial independent fixes pushed ahead as separate patches. MUCH longer commit message :) [Tvrtko Ursulin] Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
2016-06-13 23:57:32 +07:00
/* Update the GuC's idea of the doorbell ID */
desc = __get_stage_desc(client);
desc->db_id = new_id;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
}
drm/i915/guc: refactor doorbell management code This patch refactors the driver's handling and tracking of doorbells, in preparation for a later one which will resolve a suspend-resume issue. There are three resources to be managed: 1. Cachelines: a single line within the client-object's page 0 is snooped by doorbell hardware for writes from the host. 2. Doorbell registers: each defines one cacheline to be snooped. 3. Bitmap: tracks which doorbell registers are in use. The doorbell setup/teardown protocol starts with: 1. Pick a cacheline: select_doorbell_cacheline() 2. Find an available doorbell register: assign_doorbell() (These values are passed to the GuC via the shared context descriptor; this part of the sequence remains unchanged). 3. Update the bitmap to reflect registers-in-use 4. Prepare the cacheline for use by setting its status to ENABLED 5. Ask the GuC to program the doorbell to snoop the cacheline and of course teardown is very similar: 6. Set the cacheline to DISABLED 7. Ask the GuC to reprogram the doorbell to stop snooping 8. Record that the doorbell is not in use. Operations 6-8 (guc_disable_doorbell(), host2guc_release_doorbell(), and release_doorbell()) were called in sequence from guc_client_free(), but are now moved into the teardown phase of the common function. Steps 4-5 (guc_init_doorbell() and host2guc_allocate_doorbell()) were similarly done as sequential steps in guc_client_alloc(), but since it turns out that we don't need to be able to do them separately they're now collected into the setup phase of the common function. The only new code (and new capability) is the block tagged /* Update the GuC's idea of the doorbell ID */ i.e. we can now *change* the doorbell register used by an existing client, whereas previously it was set once for the entire lifetime of the client. We will use this new feature in the next patch. v2: Trivial independent fixes pushed ahead as separate patches. MUCH longer commit message :) [Tvrtko Ursulin] Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
2016-06-13 23:57:32 +07:00
static struct guc_doorbell_info *__get_doorbell(struct intel_guc_client *client)
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
{
return client->vaddr + client->doorbell_offset;
}
static bool __doorbell_valid(struct intel_guc *guc, u16 db_id)
{
struct drm_i915_private *dev_priv = guc_to_i915(guc);
GEM_BUG_ON(db_id >= GUC_NUM_DOORBELLS);
return I915_READ(GEN8_DRBREGL(db_id)) & GEN8_DRB_VALID;
}
static void __init_doorbell(struct intel_guc_client *client)
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
{
struct guc_doorbell_info *doorbell;
doorbell = __get_doorbell(client);
drm/i915/guc: refactor doorbell management code This patch refactors the driver's handling and tracking of doorbells, in preparation for a later one which will resolve a suspend-resume issue. There are three resources to be managed: 1. Cachelines: a single line within the client-object's page 0 is snooped by doorbell hardware for writes from the host. 2. Doorbell registers: each defines one cacheline to be snooped. 3. Bitmap: tracks which doorbell registers are in use. The doorbell setup/teardown protocol starts with: 1. Pick a cacheline: select_doorbell_cacheline() 2. Find an available doorbell register: assign_doorbell() (These values are passed to the GuC via the shared context descriptor; this part of the sequence remains unchanged). 3. Update the bitmap to reflect registers-in-use 4. Prepare the cacheline for use by setting its status to ENABLED 5. Ask the GuC to program the doorbell to snoop the cacheline and of course teardown is very similar: 6. Set the cacheline to DISABLED 7. Ask the GuC to reprogram the doorbell to stop snooping 8. Record that the doorbell is not in use. Operations 6-8 (guc_disable_doorbell(), host2guc_release_doorbell(), and release_doorbell()) were called in sequence from guc_client_free(), but are now moved into the teardown phase of the common function. Steps 4-5 (guc_init_doorbell() and host2guc_allocate_doorbell()) were similarly done as sequential steps in guc_client_alloc(), but since it turns out that we don't need to be able to do them separately they're now collected into the setup phase of the common function. The only new code (and new capability) is the block tagged /* Update the GuC's idea of the doorbell ID */ i.e. we can now *change* the doorbell register used by an existing client, whereas previously it was set once for the entire lifetime of the client. We will use this new feature in the next patch. v2: Trivial independent fixes pushed ahead as separate patches. MUCH longer commit message :) [Tvrtko Ursulin] Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
2016-06-13 23:57:32 +07:00
doorbell->db_status = GUC_DOORBELL_ENABLED;
doorbell->cookie = 0;
drm/i915/guc: refactor doorbell management code This patch refactors the driver's handling and tracking of doorbells, in preparation for a later one which will resolve a suspend-resume issue. There are three resources to be managed: 1. Cachelines: a single line within the client-object's page 0 is snooped by doorbell hardware for writes from the host. 2. Doorbell registers: each defines one cacheline to be snooped. 3. Bitmap: tracks which doorbell registers are in use. The doorbell setup/teardown protocol starts with: 1. Pick a cacheline: select_doorbell_cacheline() 2. Find an available doorbell register: assign_doorbell() (These values are passed to the GuC via the shared context descriptor; this part of the sequence remains unchanged). 3. Update the bitmap to reflect registers-in-use 4. Prepare the cacheline for use by setting its status to ENABLED 5. Ask the GuC to program the doorbell to snoop the cacheline and of course teardown is very similar: 6. Set the cacheline to DISABLED 7. Ask the GuC to reprogram the doorbell to stop snooping 8. Record that the doorbell is not in use. Operations 6-8 (guc_disable_doorbell(), host2guc_release_doorbell(), and release_doorbell()) were called in sequence from guc_client_free(), but are now moved into the teardown phase of the common function. Steps 4-5 (guc_init_doorbell() and host2guc_allocate_doorbell()) were similarly done as sequential steps in guc_client_alloc(), but since it turns out that we don't need to be able to do them separately they're now collected into the setup phase of the common function. The only new code (and new capability) is the block tagged /* Update the GuC's idea of the doorbell ID */ i.e. we can now *change* the doorbell register used by an existing client, whereas previously it was set once for the entire lifetime of the client. We will use this new feature in the next patch. v2: Trivial independent fixes pushed ahead as separate patches. MUCH longer commit message :) [Tvrtko Ursulin] Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
2016-06-13 23:57:32 +07:00
}
static void __fini_doorbell(struct intel_guc_client *client)
{
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
struct guc_doorbell_info *doorbell;
u16 db_id = client->doorbell_id;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
doorbell = __get_doorbell(client);
doorbell->db_status = GUC_DOORBELL_DISABLED;
/* Doorbell release flow requires that we wait for GEN8_DRB_VALID bit
* to go to zero after updating db_status before we call the GuC to
* release the doorbell
*/
if (wait_for_us(!__doorbell_valid(client->guc, db_id), 10))
WARN_ONCE(true, "Doorbell never became invalid after disable\n");
}
static int create_doorbell(struct intel_guc_client *client)
{
int ret;
if (WARN_ON(!has_doorbell(client)))
return -ENODEV; /* internal setup error, should never happen */
__update_doorbell_desc(client, client->doorbell_id);
__init_doorbell(client);
ret = __guc_allocate_doorbell(client->guc, client->stage_id);
if (ret) {
__fini_doorbell(client);
__update_doorbell_desc(client, GUC_DOORBELL_INVALID);
DRM_DEBUG_DRIVER("Couldn't create client %u doorbell: %d\n",
client->stage_id, ret);
return ret;
}
return 0;
}
static int destroy_doorbell(struct intel_guc_client *client)
{
int ret;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
GEM_BUG_ON(!has_doorbell(client));
__fini_doorbell(client);
ret = __guc_deallocate_doorbell(client->guc, client->stage_id);
if (ret)
DRM_ERROR("Couldn't destroy client %u doorbell: %d\n",
client->stage_id, ret);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
__update_doorbell_desc(client, GUC_DOORBELL_INVALID);
return ret;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
}
static unsigned long __select_cacheline(struct intel_guc *guc)
{
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
unsigned long offset;
/* Doorbell uses a single cache line within a page */
offset = offset_in_page(guc->db_cacheline);
/* Moving to next cache line to reduce contention */
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
guc->db_cacheline += cache_line_size();
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
DRM_DEBUG_DRIVER("reserved cacheline 0x%lx, next 0x%x, linesize %u\n",
offset, guc->db_cacheline, cache_line_size());
return offset;
}
static inline struct guc_process_desc *
__get_process_desc(struct intel_guc_client *client)
{
return client->vaddr + client->proc_desc_offset;
}
/*
* Initialise the process descriptor shared with the GuC firmware.
*/
static void guc_proc_desc_init(struct intel_guc_client *client)
{
struct guc_process_desc *desc;
desc = memset(__get_process_desc(client), 0, sizeof(*desc));
/*
* XXX: pDoorbell and WQVBaseAddress are pointers in process address
* space for ring3 clients (set them as in mmap_ioctl) or kernel
* space for kernel clients (map on demand instead? May make debug
* easier to have it mapped).
*/
desc->wq_base_addr = 0;
desc->db_base_addr = 0;
desc->stage_id = client->stage_id;
desc->wq_size_bytes = GUC_WQ_SIZE;
desc->wq_status = WQ_STATUS_ACTIVE;
desc->priority = client->priority;
}
static void guc_proc_desc_fini(struct intel_guc_client *client)
{
struct guc_process_desc *desc;
desc = __get_process_desc(client);
memset(desc, 0, sizeof(*desc));
}
static int guc_stage_desc_pool_create(struct intel_guc *guc)
{
struct i915_vma *vma;
void *vaddr;
vma = intel_guc_allocate_vma(guc,
PAGE_ALIGN(sizeof(struct guc_stage_desc) *
GUC_MAX_STAGE_DESCRIPTORS));
if (IS_ERR(vma))
return PTR_ERR(vma);
vaddr = i915_gem_object_pin_map(vma->obj, I915_MAP_WB);
if (IS_ERR(vaddr)) {
i915_vma_unpin_and_release(&vma, 0);
return PTR_ERR(vaddr);
}
guc->stage_desc_pool = vma;
guc->stage_desc_pool_vaddr = vaddr;
ida_init(&guc->stage_ids);
return 0;
}
static void guc_stage_desc_pool_destroy(struct intel_guc *guc)
{
ida_destroy(&guc->stage_ids);
i915_vma_unpin_and_release(&guc->stage_desc_pool, I915_VMA_RELEASE_MAP);
}
/*
* Initialise/clear the stage descriptor shared with the GuC firmware.
*
* This descriptor tells the GuC where (in GGTT space) to find the important
* data structures relating to this client (doorbell, process descriptor,
* write queue, etc).
*/
static void guc_stage_desc_init(struct intel_guc_client *client)
{
struct intel_guc *guc = client->guc;
struct drm_i915_private *dev_priv = guc_to_i915(guc);
struct intel_engine_cs *engine;
struct i915_gem_context *ctx = client->owner;
struct guc_stage_desc *desc;
unsigned int tmp;
u32 gfx_addr;
desc = __get_stage_desc(client);
memset(desc, 0, sizeof(*desc));
desc->attribute = GUC_STAGE_DESC_ATTR_ACTIVE |
GUC_STAGE_DESC_ATTR_KERNEL;
if (is_high_priority(client))
desc->attribute |= GUC_STAGE_DESC_ATTR_PREEMPT;
desc->stage_id = client->stage_id;
desc->priority = client->priority;
desc->db_id = client->doorbell_id;
for_each_engine_masked(engine, dev_priv, client->engines, tmp) {
struct intel_context *ce = to_intel_context(ctx, engine);
u32 guc_engine_id = engine->guc_id;
struct guc_execlist_context *lrc = &desc->lrc[guc_engine_id];
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
/* TODO: We have a design issue to be solved here. Only when we
* receive the first batch, we know which engine is used by the
* user. But here GuC expects the lrc and ring to be pinned. It
* is not an issue for default context, which is the only one
* for now who owns a GuC client. But for future owner of GuC
* client, need to make sure lrc is pinned prior to enter here.
*/
if (!ce->state)
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
break; /* XXX: continue? */
/*
* XXX: When this is a GUC_STAGE_DESC_ATTR_KERNEL client (proxy
* submission or, in other words, not using a direct submission
* model) the KMD's LRCA is not used for any work submission.
* Instead, the GuC uses the LRCA of the user mode context (see
* guc_add_request below).
*/
lrc->context_desc = lower_32_bits(ce->lrc_desc);
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
/* The state page is after PPHWSP */
drm/i915/guc: Rename guc_ggtt_offset to intel_guc_ggtt_offset GuC related exported functions should start with "intel_guc_" prefix and pass intel_guc as the first parameter since its GuC related. Current guc_ggtt_offset() failed to follow this code convention and this is a problem for future patches that needs to access intel_guc data to verify the GGTT offset against the GuC WOPCM top. This patch renames the guc_ggtt_offset to intel_guc_ggtt_offset and updates the related code to pass intel_guc pointer to this function call, so that we can have a unified coding style for GuC code and also enable the future patches to get GuC related data from intel_guc to do the offset verification. Meanwhile, this patch also moves the GUC_GGTT_TOP from intel_guc_regs.h to intel_guc.h since it is not GuC register related definition. v8: - Fixed coding style issues and moved GUC_GGTT_TOP to intel_guc.h (Sagar) - Updated commit message to explain to reason and motivation to add intel_guc as the first parameter of intel_guc_ggtt_offset (Chris) v9: - Fixed code alignment issue due to line break (Chris) v10: - Removed unnecessary comments, redundant code and avoided reuse variable to avoid potential issues (Joonas) v13: - Updated the ordering of s-o-b/cc/r-b tags (Sagar) Signed-off-by: Jackie Li <yaodong.li@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Sagar Arun Kamble <sagar.a.kamble@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Sagar Arun Kamble <sagar.a.kamble@intel.com> (v8) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v9) Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com> (v11) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v12) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/1520987574-19351-1-git-send-email-yaodong.li@intel.com
2018-03-14 07:32:49 +07:00
lrc->ring_lrca = intel_guc_ggtt_offset(guc, ce->state) +
LRC_STATE_PN * PAGE_SIZE;
/* XXX: In direct submission, the GuC wants the HW context id
* here. In proxy submission, it wants the stage id
*/
lrc->context_id = (client->stage_id << GUC_ELC_CTXID_OFFSET) |
(guc_engine_id << GUC_ELC_ENGINE_OFFSET);
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
drm/i915/guc: Rename guc_ggtt_offset to intel_guc_ggtt_offset GuC related exported functions should start with "intel_guc_" prefix and pass intel_guc as the first parameter since its GuC related. Current guc_ggtt_offset() failed to follow this code convention and this is a problem for future patches that needs to access intel_guc data to verify the GGTT offset against the GuC WOPCM top. This patch renames the guc_ggtt_offset to intel_guc_ggtt_offset and updates the related code to pass intel_guc pointer to this function call, so that we can have a unified coding style for GuC code and also enable the future patches to get GuC related data from intel_guc to do the offset verification. Meanwhile, this patch also moves the GUC_GGTT_TOP from intel_guc_regs.h to intel_guc.h since it is not GuC register related definition. v8: - Fixed coding style issues and moved GUC_GGTT_TOP to intel_guc.h (Sagar) - Updated commit message to explain to reason and motivation to add intel_guc as the first parameter of intel_guc_ggtt_offset (Chris) v9: - Fixed code alignment issue due to line break (Chris) v10: - Removed unnecessary comments, redundant code and avoided reuse variable to avoid potential issues (Joonas) v13: - Updated the ordering of s-o-b/cc/r-b tags (Sagar) Signed-off-by: Jackie Li <yaodong.li@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Sagar Arun Kamble <sagar.a.kamble@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Sagar Arun Kamble <sagar.a.kamble@intel.com> (v8) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v9) Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com> (v11) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v12) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/1520987574-19351-1-git-send-email-yaodong.li@intel.com
2018-03-14 07:32:49 +07:00
lrc->ring_begin = intel_guc_ggtt_offset(guc, ce->ring->vma);
lrc->ring_end = lrc->ring_begin + ce->ring->size - 1;
lrc->ring_next_free_location = lrc->ring_begin;
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
lrc->ring_current_tail_pointer_value = 0;
desc->engines_used |= (1 << guc_engine_id);
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
}
DRM_DEBUG_DRIVER("Host engines 0x%x => GuC engines used 0x%x\n",
client->engines, desc->engines_used);
WARN_ON(desc->engines_used == 0);
drm/i915: Integrate GuC-based command submission GuC-based submission is mostly the same as execlist mode, up to intel_logical_ring_advance_and_submit(), where the context being dispatched would be added to the execlist queue; at this point we submit the context to the GuC backend instead. There are, however, a few other changes also required, notably: 1. Contexts must be pinned at GGTT addresses accessible by the GuC i.e. NOT in the range [0..WOPCM_SIZE), so we have to add the PIN_OFFSET_BIAS flag to the relevant GGTT-pinning calls. 2. The GuC's TLB must be invalidated after a context is pinned at a new GGTT address. 3. GuC firmware uses the one page before Ring Context as shared data. Therefore, whenever driver wants to get base address of LRC, we will offset one page for it. LRC_PPHWSP_PN is defined as the page number of LRCA. 4. In the work queue used to pass requests to the GuC, the GuC firmware requires the ring-tail-offset to be represented as an 11-bit value, expressed in QWords. Therefore, the ringbuffer size must be reduced to the representable range (4 pages). v2: Defer adding #defines until needed [Chris Wilson] Rationalise type declarations [Chris Wilson] v4: Squashed kerneldoc patch into here [Daniel Vetter] v5: Update request->tail in code common to both GuC and execlist modes. Add a private version of lr_context_update(), as sharing the execlist version leads to race conditions when the CPU and the GuC both update TAIL in the context image. Conversion of error-captured HWS page to string must account for offset from start of object to actual HWS (LRC_PPHWSP_PN). Issue: VIZ-4884 Signed-off-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Reviewed-by: Tom O'Rourke <Tom.O'Rourke@intel.com> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2015-08-12 21:43:43 +07:00
/*
* The doorbell, process descriptor, and workqueue are all parts
* of the client object, which the GuC will reference via the GGTT
*/
drm/i915/guc: Rename guc_ggtt_offset to intel_guc_ggtt_offset GuC related exported functions should start with "intel_guc_" prefix and pass intel_guc as the first parameter since its GuC related. Current guc_ggtt_offset() failed to follow this code convention and this is a problem for future patches that needs to access intel_guc data to verify the GGTT offset against the GuC WOPCM top. This patch renames the guc_ggtt_offset to intel_guc_ggtt_offset and updates the related code to pass intel_guc pointer to this function call, so that we can have a unified coding style for GuC code and also enable the future patches to get GuC related data from intel_guc to do the offset verification. Meanwhile, this patch also moves the GUC_GGTT_TOP from intel_guc_regs.h to intel_guc.h since it is not GuC register related definition. v8: - Fixed coding style issues and moved GUC_GGTT_TOP to intel_guc.h (Sagar) - Updated commit message to explain to reason and motivation to add intel_guc as the first parameter of intel_guc_ggtt_offset (Chris) v9: - Fixed code alignment issue due to line break (Chris) v10: - Removed unnecessary comments, redundant code and avoided reuse variable to avoid potential issues (Joonas) v13: - Updated the ordering of s-o-b/cc/r-b tags (Sagar) Signed-off-by: Jackie Li <yaodong.li@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Sagar Arun Kamble <sagar.a.kamble@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Sagar Arun Kamble <sagar.a.kamble@intel.com> (v8) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v9) Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com> (v11) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v12) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/1520987574-19351-1-git-send-email-yaodong.li@intel.com
2018-03-14 07:32:49 +07:00
gfx_addr = intel_guc_ggtt_offset(guc, client->vma);
desc->db_trigger_phy = sg_dma_address(client->vma->pages->sgl) +
client->doorbell_offset;
desc->db_trigger_cpu = ptr_to_u64(__get_doorbell(client));
desc->db_trigger_uk = gfx_addr + client->doorbell_offset;
desc->process_desc = gfx_addr + client->proc_desc_offset;
desc->wq_addr = gfx_addr + GUC_DB_SIZE;
desc->wq_size = GUC_WQ_SIZE;
desc->desc_private = ptr_to_u64(client);
}
static void guc_stage_desc_fini(struct intel_guc_client *client)
{
struct guc_stage_desc *desc;
desc = __get_stage_desc(client);
memset(desc, 0, sizeof(*desc));
}
/* Construct a Work Item and append it to the GuC's Work Queue */
static void guc_wq_item_append(struct intel_guc_client *client,
u32 target_engine, u32 context_desc,
u32 ring_tail, u32 fence_id)
{
/* wqi_len is in DWords, and does not include the one-word header */
const size_t wqi_size = sizeof(struct guc_wq_item);
const u32 wqi_len = wqi_size / sizeof(u32) - 1;
struct guc_process_desc *desc = __get_process_desc(client);
struct guc_wq_item *wqi;
u32 wq_off;
lockdep_assert_held(&client->wq_lock);
/* For now workqueue item is 4 DWs; workqueue buffer is 2 pages. So we
* should not have the case where structure wqi is across page, neither
* wrapped to the beginning. This simplifies the implementation below.
*
* XXX: if not the case, we need save data to a temp wqi and copy it to
* workqueue buffer dw by dw.
*/
BUILD_BUG_ON(wqi_size != 16);
/* We expect the WQ to be active if we're appending items to it */
GEM_BUG_ON(desc->wq_status != WQ_STATUS_ACTIVE);
/* Free space is guaranteed. */
wq_off = READ_ONCE(desc->tail);
GEM_BUG_ON(CIRC_SPACE(wq_off, READ_ONCE(desc->head),
GUC_WQ_SIZE) < wqi_size);
GEM_BUG_ON(wq_off & (wqi_size - 1));
/* WQ starts from the page after doorbell / process_desc */
wqi = client->vaddr + wq_off + GUC_DB_SIZE;
if (I915_SELFTEST_ONLY(client->use_nop_wqi)) {
wqi->header = WQ_TYPE_NOOP | (wqi_len << WQ_LEN_SHIFT);
} else {
/* Now fill in the 4-word work queue item */
wqi->header = WQ_TYPE_INORDER |
(wqi_len << WQ_LEN_SHIFT) |
(target_engine << WQ_TARGET_SHIFT) |
WQ_NO_WCFLUSH_WAIT;
wqi->context_desc = context_desc;
wqi->submit_element_info = ring_tail << WQ_RING_TAIL_SHIFT;
GEM_BUG_ON(ring_tail > WQ_RING_TAIL_MAX);
wqi->fence_id = fence_id;
}
/* Make the update visible to GuC */
WRITE_ONCE(desc->tail, (wq_off + wqi_size) & (GUC_WQ_SIZE - 1));
}
static void guc_ring_doorbell(struct intel_guc_client *client)
{
struct guc_doorbell_info *db;
u32 cookie;
lockdep_assert_held(&client->wq_lock);
/* pointer of current doorbell cacheline */
db = __get_doorbell(client);
/*
* We're not expecting the doorbell cookie to change behind our back,
* we also need to treat 0 as a reserved value.
*/
cookie = READ_ONCE(db->cookie);
WARN_ON_ONCE(xchg(&db->cookie, cookie + 1 ?: cookie + 2) != cookie);
/* XXX: doorbell was lost and need to acquire it again */
GEM_BUG_ON(db->db_status != GUC_DOORBELL_ENABLED);
}
static void guc_add_request(struct intel_guc *guc, struct i915_request *rq)
{
struct intel_guc_client *client = guc->execbuf_client;
struct intel_engine_cs *engine = rq->engine;
u32 ctx_desc = lower_32_bits(rq->hw_context->lrc_desc);
u32 ring_tail = intel_ring_set_tail(rq->ring, rq->tail) / sizeof(u64);
spin_lock(&client->wq_lock);
guc_wq_item_append(client, engine->guc_id, ctx_desc,
ring_tail, rq->global_seqno);
guc_ring_doorbell(client);
client->submissions[engine->id] += 1;
spin_unlock(&client->wq_lock);
}
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
/*
* When we're doing submissions using regular execlists backend, writing to
* ELSP from CPU side is enough to make sure that writes to ringbuffer pages
* pinned in mappable aperture portion of GGTT are visible to command streamer.
* Writes done by GuC on our behalf are not guaranteeing such ordering,
* therefore, to ensure the flush, we're issuing a POSTING READ.
*/
static void flush_ggtt_writes(struct i915_vma *vma)
{
struct drm_i915_private *dev_priv = vma->vm->i915;
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
if (i915_vma_is_map_and_fenceable(vma))
POSTING_READ_FW(GUC_STATUS);
}
static void inject_preempt_context(struct work_struct *work)
{
struct guc_preempt_work *preempt_work =
container_of(work, typeof(*preempt_work), work);
struct intel_engine_cs *engine = preempt_work->engine;
struct intel_guc *guc = container_of(preempt_work, typeof(*guc),
preempt_work[engine->id]);
struct intel_guc_client *client = guc->preempt_client;
struct guc_stage_desc *stage_desc = __get_stage_desc(client);
struct intel_context *ce = to_intel_context(client->owner, engine);
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
u32 data[7];
if (!ce->ring->emit) { /* recreate upon load/resume */
u32 addr = intel_hws_preempt_done_address(engine);
u32 *cs;
cs = ce->ring->vaddr;
if (engine->id == RCS) {
cs = gen8_emit_ggtt_write_rcs(cs,
GUC_PREEMPT_FINISHED,
addr);
} else {
cs = gen8_emit_ggtt_write(cs,
GUC_PREEMPT_FINISHED,
addr);
*cs++ = MI_NOOP;
*cs++ = MI_NOOP;
}
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_NOOP;
ce->ring->emit = GUC_PREEMPT_BREADCRUMB_BYTES;
GEM_BUG_ON((void *)cs - ce->ring->vaddr != ce->ring->emit);
flush_ggtt_writes(ce->ring->vma);
}
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
spin_lock_irq(&client->wq_lock);
guc_wq_item_append(client, engine->guc_id, lower_32_bits(ce->lrc_desc),
GUC_PREEMPT_BREADCRUMB_BYTES / sizeof(u64), 0);
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
spin_unlock_irq(&client->wq_lock);
/*
* If GuC firmware performs an engine reset while that engine had
* a preemption pending, it will set the terminated attribute bit
* on our preemption stage descriptor. GuC firmware retains all
* pending work items for a high-priority GuC client, unlike the
* normal-priority GuC client where work items are dropped. It
* wants to make sure the preempt-to-idle work doesn't run when
* scheduling resumes, and uses this bit to inform its scheduler
* and presumably us as well. Our job is to clear it for the next
* preemption after reset, otherwise that and future preemptions
* will never complete. We'll just clear it every time.
*/
stage_desc->attribute &= ~GUC_STAGE_DESC_ATTR_TERMINATED;
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
data[0] = INTEL_GUC_ACTION_REQUEST_PREEMPTION;
data[1] = client->stage_id;
data[2] = INTEL_GUC_PREEMPT_OPTION_DROP_WORK_Q |
INTEL_GUC_PREEMPT_OPTION_DROP_SUBMIT_Q;
data[3] = engine->guc_id;
data[4] = guc->execbuf_client->priority;
data[5] = guc->execbuf_client->stage_id;
drm/i915/guc: Rename guc_ggtt_offset to intel_guc_ggtt_offset GuC related exported functions should start with "intel_guc_" prefix and pass intel_guc as the first parameter since its GuC related. Current guc_ggtt_offset() failed to follow this code convention and this is a problem for future patches that needs to access intel_guc data to verify the GGTT offset against the GuC WOPCM top. This patch renames the guc_ggtt_offset to intel_guc_ggtt_offset and updates the related code to pass intel_guc pointer to this function call, so that we can have a unified coding style for GuC code and also enable the future patches to get GuC related data from intel_guc to do the offset verification. Meanwhile, this patch also moves the GUC_GGTT_TOP from intel_guc_regs.h to intel_guc.h since it is not GuC register related definition. v8: - Fixed coding style issues and moved GUC_GGTT_TOP to intel_guc.h (Sagar) - Updated commit message to explain to reason and motivation to add intel_guc as the first parameter of intel_guc_ggtt_offset (Chris) v9: - Fixed code alignment issue due to line break (Chris) v10: - Removed unnecessary comments, redundant code and avoided reuse variable to avoid potential issues (Joonas) v13: - Updated the ordering of s-o-b/cc/r-b tags (Sagar) Signed-off-by: Jackie Li <yaodong.li@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Sagar Arun Kamble <sagar.a.kamble@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Sagar Arun Kamble <sagar.a.kamble@intel.com> (v8) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v9) Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com> (v11) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> (v12) Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/1520987574-19351-1-git-send-email-yaodong.li@intel.com
2018-03-14 07:32:49 +07:00
data[6] = intel_guc_ggtt_offset(guc, guc->shared_data);
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
if (WARN_ON(intel_guc_send(guc, data, ARRAY_SIZE(data)))) {
execlists_clear_active(&engine->execlists,
EXECLISTS_ACTIVE_PREEMPT);
tasklet_schedule(&engine->execlists.tasklet);
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
}
}
/*
* We're using user interrupt and HWSP value to mark that preemption has
* finished and GPU is idle. Normally, we could unwind and continue similar to
* execlists submission path. Unfortunately, with GuC we also need to wait for
* it to finish its own postprocessing, before attempting to submit. Otherwise
* GuC may silently ignore our submissions, and thus we risk losing request at
* best, executing out-of-order and causing kernel panic at worst.
*/
#define GUC_PREEMPT_POSTPROCESS_DELAY_MS 10
static void wait_for_guc_preempt_report(struct intel_engine_cs *engine)
{
struct intel_guc *guc = &engine->i915->guc;
struct guc_shared_ctx_data *data = guc->shared_data_vaddr;
struct guc_ctx_report *report =
&data->preempt_ctx_report[engine->guc_id];
WARN_ON(wait_for_atomic(report->report_return_status ==
INTEL_GUC_REPORT_STATUS_COMPLETE,
GUC_PREEMPT_POSTPROCESS_DELAY_MS));
/*
* GuC is expecting that we're also going to clear the affected context
* counter, let's also reset the return status to not depend on GuC
* resetting it after recieving another preempt action
*/
report->affected_count = 0;
report->report_return_status = INTEL_GUC_REPORT_STATUS_UNKNOWN;
}
static void complete_preempt_context(struct intel_engine_cs *engine)
{
struct intel_engine_execlists *execlists = &engine->execlists;
GEM_BUG_ON(!execlists_is_active(execlists, EXECLISTS_ACTIVE_PREEMPT));
if (inject_preempt_hang(execlists))
return;
execlists_cancel_port_requests(execlists);
execlists_unwind_incomplete_requests(execlists);
wait_for_guc_preempt_report(engine);
intel_write_status_page(engine, I915_GEM_HWS_PREEMPT_INDEX, 0);
}
/**
* guc_submit() - Submit commands through GuC
* @engine: engine associated with the commands
*
* The only error here arises if the doorbell hardware isn't functioning
* as expected, which really shouln't happen.
*/
static void guc_submit(struct intel_engine_cs *engine)
{
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
struct intel_guc *guc = &engine->i915->guc;
struct intel_engine_execlists * const execlists = &engine->execlists;
struct execlist_port *port = execlists->port;
unsigned int n;
for (n = 0; n < execlists_num_ports(execlists); n++) {
struct i915_request *rq;
unsigned int count;
drm/i915/guc: WA to address the Ringbuffer coherency issue Driver accesses the ringbuffer pages, via GMADR BAR, if the pages are pinned in mappable aperture portion of GGTT and for ringbuffer pages allocated from Stolen memory, access can only be done through GMADR BAR. In case of GuC based submission, updates done in ringbuffer via GMADR may not get committed to memory by the time the Command streamer starts reading them, resulting in fetching of stale data. For Host based submission, such problem is not there as the write to Ring Tail or ELSP register happens from the Host side prior to submission. Access to any GFX register from CPU side goes to GTTMMADR BAR and Hw already enforces the ordering between outstanding GMADR writes & new GTTMADR access. MMIO writes from GuC side do not go to GTTMMADR BAR as GuC communication to registers within GT is contained within GT, so ordering is not enforced resulting in a race, which can manifest in form of a hang. To ensure the flush of in-flight GMADR writes, a POSTING READ is done to GuC register prior to doorbell ring. There is already a similar WA in i915_gem_object_flush_gtt_write_domain(), which takes care of GMADR writes from User space to GEM buffers, but not the ringbuffer writes from KMD. This WA is needed on all recent HW. v2: - Use POSTING_READ_FW instead of POSTING_READ as GuC register do not lie in any forcewake domain range and so the overhead of spinlock & search in the forcewake table is avoidable. (Chris) Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Sagar Arun Kamble <sagar.a.kamble@intel.com> Signed-off-by: Akash Goel <akash.goel@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: http://patchwork.freedesktop.org/patch/msgid/1477413323-1880-1-git-send-email-akash.goel@intel.com
2016-10-25 23:35:23 +07:00
rq = port_unpack(&port[n], &count);
if (rq && count == 0) {
port_set(&port[n], port_pack(rq, ++count));
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
flush_ggtt_writes(rq->ring->vma);
guc_add_request(guc, rq);
}
}
drm/i915/guc: Split hw submission for replay after GPU reset Something I missed before sending off the partial series was that the non-scheduler guc reset path was broken (in the full series, this is pushed to the execlists reset handler). The issue is that after a reset, we have to refill the GuC workqueues, which we do by resubmitting the requests. However, if we already have submitted them, the fences within them have already been used and triggering them again is an error. Instead, just repopulate the guc workqueue. [ 115.858560] [IGT] gem_busy: starting subtest hang-render [ 135.839867] [drm] GPU HANG: ecode 9:0:0xe757fefe, in gem_busy [1716], reason: Hang on render ring, action: reset [ 135.839902] drm/i915: Resetting chip after gpu hang [ 135.839957] [drm] RC6 on [ 135.858351] ------------[ cut here ]------------ [ 135.858357] WARNING: CPU: 2 PID: 45 at drivers/gpu/drm/i915/i915_sw_fence.c:108 i915_sw_fence_complete+0x25/0x30 [ 135.858357] Modules linked in: rfcomm bnep binfmt_misc nls_iso8859_1 input_leds snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core btusb btrtl snd_hwdep snd_pcm 8250_dw snd_seq_midi hid_lenovo snd_seq_midi_event snd_rawmidi iwlwifi x86_pkg_temp_thermal coretemp snd_seq crct10dif_pclmul snd_seq_device hci_uart snd_timer crc32_pclmul ghash_clmulni_intel idma64 aesni_intel virt_dma btbcm snd btqca aes_x86_64 btintel lrw cfg80211 bluetooth gf128mul glue_helper ablk_helper cryptd soundcore intel_lpss_pci intel_pch_thermal intel_lpss_acpi intel_lpss acpi_als mfd_core kfifo_buf acpi_pad industrialio autofs4 hid_plantronics usbhid dm_mirror dm_region_hash dm_log sdhci_pci ahci sdhci libahci i2c_hid hid [ 135.858389] CPU: 2 PID: 45 Comm: kworker/2:1 Tainted: G W 4.9.0-rc4+ #238 [ 135.858389] Hardware name: /NUC6i3SYB, BIOS SYSKLi35.86A.0024.2015.1027.2142 10/27/2015 [ 135.858392] Workqueue: events_long i915_hangcheck_elapsed [ 135.858394] ffffc900001bf9b8 ffffffff812bb238 0000000000000000 0000000000000000 [ 135.858396] ffffc900001bf9f8 ffffffff8104f621 0000006c00000000 ffff8808296137f8 [ 135.858398] 0000000000000a00 ffff8808457a0000 ffff880845764e60 ffff880845760000 [ 135.858399] Call Trace: [ 135.858403] [<ffffffff812bb238>] dump_stack+0x4d/0x65 [ 135.858405] [<ffffffff8104f621>] __warn+0xc1/0xe0 [ 135.858406] [<ffffffff8104f748>] warn_slowpath_null+0x18/0x20 [ 135.858408] [<ffffffff813f8c15>] i915_sw_fence_complete+0x25/0x30 [ 135.858410] [<ffffffff813f8fad>] i915_sw_fence_commit+0xd/0x30 [ 135.858412] [<ffffffff8142e591>] __i915_gem_request_submit+0xe1/0xf0 [ 135.858413] [<ffffffff8142e5c8>] i915_gem_request_submit+0x28/0x40 [ 135.858415] [<ffffffff814433e7>] i915_guc_submit+0x47/0x210 [ 135.858417] [<ffffffff81443e98>] i915_guc_submission_enable+0x468/0x540 [ 135.858419] [<ffffffff81442495>] intel_guc_setup+0x715/0x810 [ 135.858421] [<ffffffff8142b6b4>] i915_gem_init_hw+0x114/0x2a0 [ 135.858423] [<ffffffff813eeaa8>] i915_reset+0xe8/0x120 [ 135.858424] [<ffffffff813f3937>] i915_reset_and_wakeup+0x157/0x180 [ 135.858426] [<ffffffff813f79db>] i915_handle_error+0x1ab/0x230 [ 135.858428] [<ffffffff812c760d>] ? scnprintf+0x4d/0x90 [ 135.858430] [<ffffffff81435985>] i915_hangcheck_elapsed+0x275/0x3d0 [ 135.858432] [<ffffffff810668cf>] process_one_work+0x12f/0x410 [ 135.858433] [<ffffffff81066bf3>] worker_thread+0x43/0x4d0 [ 135.858435] [<ffffffff81066bb0>] ? process_one_work+0x410/0x410 [ 135.858436] [<ffffffff81066bb0>] ? process_one_work+0x410/0x410 [ 135.858438] [<ffffffff8106bbb4>] kthread+0xd4/0xf0 [ 135.858440] [<ffffffff8106bae0>] ? kthread_park+0x60/0x60 v2: Only resubmit submitted requests v3: Don't forget the pending requests have reserved space. Fixes: d55ac5bf97c6 ("drm/i915: Defer transfer onto execution timeline to actual hw submission") Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20161129121024.22650-6-chris@chris-wilson.co.uk
2016-11-29 19:10:24 +07:00
}
static void port_assign(struct execlist_port *port, struct i915_request *rq)
{
GEM_BUG_ON(port_isset(port));
port_set(port, i915_request_get(rq));
}
static inline int rq_prio(const struct i915_request *rq)
{
return rq->sched.attr.priority;
}
static inline int port_prio(const struct execlist_port *port)
{
return rq_prio(port_request(port));
}
static bool __guc_dequeue(struct intel_engine_cs *engine)
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
{
struct intel_engine_execlists * const execlists = &engine->execlists;
struct execlist_port *port = execlists->port;
struct i915_request *last = NULL;
const struct execlist_port * const last_port =
&execlists->port[execlists->port_mask];
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
bool submit = false;
struct rb_node *rb;
lockdep_assert_held(&engine->timeline.lock);
if (port_isset(port)) {
if (intel_engine_has_preemption(engine)) {
struct guc_preempt_work *preempt_work =
&engine->i915->guc.preempt_work[engine->id];
int prio = execlists->queue_priority;
if (__execlists_need_preempt(prio, port_prio(port))) {
execlists_set_active(execlists,
EXECLISTS_ACTIVE_PREEMPT);
queue_work(engine->i915->guc.preempt_wq,
&preempt_work->work);
return false;
}
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
}
port++;
if (port_isset(port))
return false;
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
}
GEM_BUG_ON(port_isset(port));
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
while ((rb = rb_first_cached(&execlists->queue))) {
struct i915_priolist *p = to_priolist(rb);
struct i915_request *rq, *rn;
int i;
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
priolist_for_each_request_consume(rq, rn, p, i) {
if (last && rq->hw_context != last->hw_context) {
if (port == last_port)
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
goto done;
if (submit)
port_assign(port, last);
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
port++;
}
list_del_init(&rq->sched.link);
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
__i915_request_submit(rq);
trace_i915_request_in(rq, port_index(port, execlists));
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
last = rq;
submit = true;
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
}
rb_erase_cached(&p->node, &execlists->queue);
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
if (p->priority != I915_PRIORITY_NORMAL)
kmem_cache_free(engine->i915->priorities, p);
}
drm/i915: Split execlist priority queue into rbtree + linked list All the requests at the same priority are executed in FIFO order. They do not need to be stored in the rbtree themselves, as they are a simple list within a level. If we move the requests at one priority into a list, we can then reduce the rbtree to the set of priorities. This should keep the height of the rbtree small, as the number of active priorities can not exceed the number of active requests and should be typically only a few. Currently, we have ~2k possible different priority levels, that may increase to allow even more fine grained selection. Allocating those in advance seems a waste (and may be impossible), so we opt for allocating upon first use, and freeing after its requests are depleted. To avoid the possibility of an allocation failure causing us to lose a request, we preallocate the default priority (0) and bump any request to that priority if we fail to allocate it the appropriate plist. Having a request (that is ready to run, so not leading to corruption) execute out-of-order is better than leaking the request (and its dependency tree) entirely. There should be a benefit to reducing execlists_dequeue() to principally using a simple list (and reducing the frequency of both rbtree iteration and balancing on erase) but for typical workloads, request coalescing should be small enough that we don't notice any change. The main gain is from improving PI calls to schedule, and the explicit list within a level should make request unwinding simpler (we just need to insert at the head of the list rather than the tail and not have to make the rbtree search more complicated). v2: Avoid use-after-free when deleting a depleted priolist v3: Michał found the solution to handling the allocation failure gracefully. If we disable all priority scheduling following the allocation failure, those requests will be executed in fifo and we will ensure that this request and its dependencies are in strict fifo (even when it doesn't realise it is only a single list). Normal scheduling is restored once we know the device is idle, until the next failure! Suggested-by: Michał Wajdeczko <michal.wajdeczko@intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170517121007.27224-8-chris@chris-wilson.co.uk
2017-05-17 19:10:03 +07:00
done:
execlists->queue_priority = rb ? to_priolist(rb)->priority : INT_MIN;
if (submit)
port_assign(port, last);
if (last)
execlists_user_begin(execlists, execlists->port);
/* We must always keep the beast fed if we have work piled up */
GEM_BUG_ON(port_isset(execlists->port) &&
!execlists_is_active(execlists, EXECLISTS_ACTIVE_USER));
GEM_BUG_ON(rb_first_cached(&execlists->queue) &&
!port_isset(execlists->port));
return submit;
}
static void guc_dequeue(struct intel_engine_cs *engine)
{
if (__guc_dequeue(engine))
guc_submit(engine);
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
}
static void guc_submission_tasklet(unsigned long data)
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
{
struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
struct intel_engine_execlists * const execlists = &engine->execlists;
struct execlist_port *port = execlists->port;
struct i915_request *rq;
unsigned long flags;
spin_lock_irqsave(&engine->timeline.lock, flags);
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
rq = port_request(port);
while (rq && i915_request_completed(rq)) {
trace_i915_request_out(rq);
i915_request_put(rq);
port = execlists_port_complete(execlists, port);
if (port_isset(port)) {
execlists_user_begin(execlists, port);
rq = port_request(port);
} else {
execlists_user_end(execlists);
rq = NULL;
}
}
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
if (execlists_is_active(execlists, EXECLISTS_ACTIVE_PREEMPT) &&
intel_read_status_page(engine, I915_GEM_HWS_PREEMPT_INDEX) ==
GUC_PREEMPT_FINISHED)
complete_preempt_context(engine);
drm/i915/guc: Preemption! With GuC Pretty similar to what we have on execlists. We're reusing most of the GEM code, however, due to GuC quirks we need a couple of extra bits. Preemption is implemented as GuC action, and actions can be pretty slow. Because of that, we're using a mutex to serialize them. Since we're requesting preemption from the tasklet, the task of creating a workitem and wrapping it in GuC action is delegated to a worker. To distinguish that preemption has finished, we're using additional piece of HWSP, and since we're not getting context switch interrupts, we're also adding a user interrupt. The fact that our special preempt context has completed unfortunately doesn't mean that we're ready to submit new work. We also need to wait for GuC to finish its own processing. v2: Don't compile out the wait for GuC, handle workqueue flush on reset, no need for ordered workqueue, put on a reviewer hat when looking at my own patches (Chris) Move struct work around in intel_guc, move user interruput outside of conditional (Michał) Keep ring around rather than chase though intel_context v3: Extract WA for flushing ggtt writes to a helper (Chris) Keep work_struct in intel_guc rather than engine (Michał) Use ordered workqueue for inject_preempt worker to avoid GuC quirks. v4: Drop now unused INTEL_GUC_PREEMPT_OPTION_IMMEDIATE (Daniele) Drop stray newlines, use container_of for intel_guc in worker, check for presence of workqueue when flushing it, rather than enable_guc_submission modparam, reorder preempt postprocessing (Chris) v5: Make wq NULL after destroying it v6: Swap struct guc_preempt_work members (Michał) Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Jeff McGee <jeff.mcgee@intel.com> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Oscar Mateo <oscar.mateo@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171026133558.19580-1-michal.winiarski@intel.com
2017-10-26 20:35:58 +07:00
if (!execlists_is_active(execlists, EXECLISTS_ACTIVE_PREEMPT))
guc_dequeue(engine);
spin_unlock_irqrestore(&engine->timeline.lock, flags);
drm/i915/scheduler: emulate a scheduler for guc This emulates execlists on top of the GuC in order to defer submission of requests to the hardware. This deferral allows time for high priority requests to gazump their way to the head of the queue, however it nerfs the GuC by converting it back into a simple execlist (where the CPU has to wake up after every request to feed new commands into the GuC). v2: Drop hack status - though iirc there is still a lockdep inversion between fence and engine->timeline->lock (which is impossible as the nesting only occurs on different fences - hopefully just requires some judicious lockdep annotation) v3: Apply lockdep nesting to enabling signaling on the request, using the pattern we already have in __i915_gem_request_submit(); v4: Replaying requests after a hang also now needs the timeline spinlock, to disable the interrupts at least v5: Hold wq lock for completeness, and emit a tracepoint for enabling signal v6: Reorder interrupt checking for a happier gcc. v7: Only signal the tasklet after a user-interrupt if using guc scheduling v8: Restore lost update of rq through the i915_guc_irq_handler (Tvrtko) v9: Avoid re-initialising the engine->irq_tasklet from inside a reset v10: Hook up the execlists-style tracepoints v11: Clear the execlists irq_posted bit after taking over the interrupt/tasklet Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20170316125619.6856-1-chris@chris-wilson.co.uk Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
2017-03-16 19:56:18 +07:00
}
static struct i915_request *
guc_reset_prepare(struct intel_engine_cs *engine)
{
struct intel_engine_execlists * const execlists = &engine->execlists;
GEM_TRACE("%s\n", engine->name);
/*
* Prevent request submission to the hardware until we have
* completed the reset in i915_gem_reset_finish(). If a request
* is completed by one engine, it may then queue a request
* to a second via its execlists->tasklet *just* as we are
* calling engine->init_hw() and also writing the ELSP.
* Turning off the execlists->tasklet until the reset is over
* prevents the race.
*/
__tasklet_disable_sync_once(&execlists->tasklet);
/*
* We're using worker to queue preemption requests from the tasklet in
* GuC submission mode.
* Even though tasklet was disabled, we may still have a worker queued.
* Let's make sure that all workers scheduled before disabling the
* tasklet are completed before continuing with the reset.
*/
if (engine->i915->guc.preempt_wq)
flush_workqueue(engine->i915->guc.preempt_wq);
return i915_gem_find_active_request(engine);
}
/*
* Everything below here is concerned with setup & teardown, and is
* therefore not part of the somewhat time-critical batch-submission
* path of guc_submit() above.
*/
/* Check that a doorbell register is in the expected state */
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
static bool doorbell_ok(struct intel_guc *guc, u16 db_id)
{
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
bool valid;
GEM_BUG_ON(db_id >= GUC_NUM_DOORBELLS);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
valid = __doorbell_valid(guc, db_id);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (test_bit(db_id, guc->doorbell_bitmap) == valid)
return true;
DRM_DEBUG_DRIVER("Doorbell %u has unexpected state: valid=%s\n",
db_id, yesno(valid));
return false;
}
static bool guc_verify_doorbells(struct intel_guc *guc)
{
bool doorbells_ok = true;
u16 db_id;
for (db_id = 0; db_id < GUC_NUM_DOORBELLS; ++db_id)
if (!doorbell_ok(guc, db_id))
doorbells_ok = false;
return doorbells_ok;
}
/**
* guc_client_alloc() - Allocate an intel_guc_client
* @dev_priv: driver private data structure
* @engines: The set of engines to enable for this client
* @priority: four levels priority _CRITICAL, _HIGH, _NORMAL and _LOW
* The kernel client to replace ExecList submission is created with
* NORMAL priority. Priority of a client for scheduler can be HIGH,
* while a preemption context can use CRITICAL.
* @ctx: the context that owns the client (we use the default render
* context)
*
* Return: An intel_guc_client object if success, else NULL.
*/
static struct intel_guc_client *
guc_client_alloc(struct drm_i915_private *dev_priv,
u32 engines,
u32 priority,
struct i915_gem_context *ctx)
{
struct intel_guc_client *client;
struct intel_guc *guc = &dev_priv->guc;
struct i915_vma *vma;
void *vaddr;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
int ret;
client = kzalloc(sizeof(*client), GFP_KERNEL);
if (!client)
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
return ERR_PTR(-ENOMEM);
client->guc = guc;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
client->owner = ctx;
client->engines = engines;
client->priority = priority;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
client->doorbell_id = GUC_DOORBELL_INVALID;
spin_lock_init(&client->wq_lock);
ret = ida_simple_get(&guc->stage_ids, 0, GUC_MAX_STAGE_DESCRIPTORS,
GFP_KERNEL);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (ret < 0)
goto err_client;
client->stage_id = ret;
/* The first page is doorbell/proc_desc. Two followed pages are wq. */
vma = intel_guc_allocate_vma(guc, GUC_DB_SIZE + GUC_WQ_SIZE);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (IS_ERR(vma)) {
ret = PTR_ERR(vma);
goto err_id;
}
drm/i915/guc: keep GuC doorbell & process descriptor mapped in kernel Don't use kmap_atomic() for doorbell & process descriptor access. This patch fixes the BUG shown below, where the thread could sleep while holding a kmap_atomic mapping. In order not to need to call kmap_atomic() in this code path, we now set up a permanent kernel mapping of the shared doorbell and process-descriptor page, and use that in all doorbell and process-descriptor related code. BUG: scheduling while atomic: gem_close_race/1941/0x00000002 Modules linked in: hid_generic usbhid i915 asix usbnet libphy mii i2c_algo_bit drm_kms_helper cfbfillrect syscopyarea cfbimgblt sysfillrect sysimgblt fb_sys_fops cfbcopyarea drm coretemp i2c_hid hid video pinctrl_sunrisepoint pinctrl_intel acpi_pad nls_iso8859_1 e1000e ptp psmouse pps_core ahci libahci CPU: 0 PID: 1941 Comm: gem_close_race Tainted: G U 4.4.0-160121+ #123 Hardware name: Intel Corporation Skylake Client platform/Skylake AIO DDR3L RVP10, BIOS SKLSE2R1.R00.X100.B01.1509220551 09/22/2015 0000000000013e40 ffff880166c27a78 ffffffff81280d02 ffff880172c13e40 ffff880166c27a88 ffffffff810c203a ffff880166c27ac8 ffffffff814ec808 ffff88016b7c6000 ffff880166c28000 00000000000f4240 0000000000000001 Call Trace: [<ffffffff81280d02>] dump_stack+0x4b/0x79 [<ffffffff810c203a>] __schedule_bug+0x41/0x4f [<ffffffff814ec808>] __schedule+0x5a8/0x690 [<ffffffff814ec927>] schedule+0x37/0x80 [<ffffffff814ef3fd>] schedule_hrtimeout_range_clock+0xad/0x130 [<ffffffff81090be0>] ? hrtimer_init+0x10/0x10 [<ffffffff814ef3f1>] ? schedule_hrtimeout_range_clock+0xa1/0x130 [<ffffffff814ef48e>] schedule_hrtimeout_range+0xe/0x10 [<ffffffff814eef9b>] usleep_range+0x3b/0x40 [<ffffffffa01ec109>] i915_guc_wq_check_space+0x119/0x210 [i915] [<ffffffffa01da47c>] intel_logical_ring_alloc_request_extras+0x5c/0x70 [i915] [<ffffffffa01cdbf1>] i915_gem_request_alloc+0x91/0x170 [i915] [<ffffffffa01c1c07>] i915_gem_do_execbuffer.isra.25+0xbc7/0x12a0 [i915] [<ffffffffa01cb785>] ? i915_gem_object_get_pages_gtt+0x225/0x3c0 [i915] [<ffffffffa01d1fb6>] ? i915_gem_pwrite_ioctl+0xd6/0x9f0 [i915] [<ffffffffa01c2e68>] i915_gem_execbuffer2+0xa8/0x250 [i915] [<ffffffffa00f65d8>] drm_ioctl+0x258/0x4f0 [drm] [<ffffffffa01c2dc0>] ? i915_gem_execbuffer+0x340/0x340 [i915] [<ffffffff8111590d>] do_vfs_ioctl+0x2cd/0x4a0 [<ffffffff8111eac2>] ? __fget+0x72/0xb0 [<ffffffff81115b1c>] SyS_ioctl+0x3c/0x70 [<ffffffff814effd7>] entry_SYSCALL_64_fastpath+0x12/0x6a ------------[ cut here ]------------ v4: Only tear down doorbell & kunmap() client object if we actually succeeded in allocating a client object (Tvrtko Ursulin) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=93847 Original-version-by: Alex Dai <yu.dai@intel.com> Signed-off-by: Dave Gordon <david.s.gordon@intel.com> Cc: Tvtrko Ursulin <tvrtko.ursulin@intel.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
2016-04-19 22:08:34 +07:00
/* We'll keep just the first (doorbell/proc) page permanently kmap'd. */
client->vma = vma;
vaddr = i915_gem_object_pin_map(vma->obj, I915_MAP_WB);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (IS_ERR(vaddr)) {
ret = PTR_ERR(vaddr);
goto err_vma;
}
client->vaddr = vaddr;
ret = reserve_doorbell(client);
if (ret)
goto err_vaddr;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
client->doorbell_offset = __select_cacheline(guc);
/*
* Since the doorbell only requires a single cacheline, we can save
* space by putting the application process descriptor in the same
* page. Use the half of the page that doesn't include the doorbell.
*/
if (client->doorbell_offset >= (GUC_DB_SIZE / 2))
client->proc_desc_offset = 0;
else
client->proc_desc_offset = (GUC_DB_SIZE / 2);
DRM_DEBUG_DRIVER("new priority %u client %p for engine(s) 0x%x: stage_id %u\n",
priority, client, client->engines, client->stage_id);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
DRM_DEBUG_DRIVER("doorbell id %u, cacheline offset 0x%lx\n",
client->doorbell_id, client->doorbell_offset);
return client;
err_vaddr:
i915_gem_object_unpin_map(client->vma->obj);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
err_vma:
i915_vma_unpin_and_release(&client->vma, 0);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
err_id:
ida_simple_remove(&guc->stage_ids, client->stage_id);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
err_client:
kfree(client);
return ERR_PTR(ret);
}
static void guc_client_free(struct intel_guc_client *client)
{
unreserve_doorbell(client);
i915_vma_unpin_and_release(&client->vma, I915_VMA_RELEASE_MAP);
ida_simple_remove(&client->guc->stage_ids, client->stage_id);
kfree(client);
}
static inline bool ctx_save_restore_disabled(struct intel_context *ce)
{
u32 sr = ce->lrc_reg_state[CTX_CONTEXT_CONTROL + 1];
#define SR_DISABLED \
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT | \
CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT)
return (sr & SR_DISABLED) == SR_DISABLED;
#undef SR_DISABLED
}
static int guc_clients_create(struct intel_guc *guc)
{
struct drm_i915_private *dev_priv = guc_to_i915(guc);
struct intel_guc_client *client;
GEM_BUG_ON(guc->execbuf_client);
GEM_BUG_ON(guc->preempt_client);
client = guc_client_alloc(dev_priv,
INTEL_INFO(dev_priv)->ring_mask,
GUC_CLIENT_PRIORITY_KMD_NORMAL,
dev_priv->kernel_context);
if (IS_ERR(client)) {
DRM_ERROR("Failed to create GuC client for submission!\n");
return PTR_ERR(client);
}
guc->execbuf_client = client;
if (dev_priv->preempt_context) {
client = guc_client_alloc(dev_priv,
INTEL_INFO(dev_priv)->ring_mask,
GUC_CLIENT_PRIORITY_KMD_HIGH,
dev_priv->preempt_context);
if (IS_ERR(client)) {
DRM_ERROR("Failed to create GuC client for preemption!\n");
guc_client_free(guc->execbuf_client);
guc->execbuf_client = NULL;
return PTR_ERR(client);
}
guc->preempt_client = client;
}
return 0;
}
static void guc_clients_destroy(struct intel_guc *guc)
{
struct intel_guc_client *client;
client = fetch_and_zero(&guc->preempt_client);
if (client)
guc_client_free(client);
client = fetch_and_zero(&guc->execbuf_client);
drm/i915/guc: Protect against NULL client dereference in error path After aborting a module load, we may try and disable guc before we have finished setting it. Long term plan is to ensure perfect onion unwind, but in the short term we want to fix the oops to re-enable drv_module_reload. [ 317.401239] BUG: unable to handle kernel NULL pointer dereference at 0000000000000030 [ 317.401279] Oops: 0000 [#1] PREEMPT SMP PTI [ 317.401294] CPU: 5 PID: 4275 Comm: drv_module_relo Tainted: G U 4.18.0-rc4-CI-CI_DRM_4476+ #1 [ 317.401317] Hardware name: System manufacturer System Product Name/Z170M-PLUS, BIOS 3610 03/29/2018 [ 317.401440] RIP: 0010:unreserve_doorbell+0x0/0x80 [i915] [ 317.401454] Code: bb e0 48 8b 35 21 4d 18 00 49 c7 c0 a8 e5 62 a0 b9 cc 00 00 00 48 c7 c2 d8 41 5f a0 48 c7 c7 c9 f6 53 a0 e8 a2 3d c2 e0 0f 0b <0f> b7 47 30 66 3d 00 01 74 20 48 8b 57 18 48 0f a3 82 40 05 00 00 [ 317.401602] RSP: 0018:ffffc900003d3da0 EFLAGS: 00010246 [ 317.401619] RAX: ffffffff8223b300 RBX: 0000000000000000 RCX: 0000000000000000 [ 317.401636] RDX: 0000001fffffffc0 RSI: ffff880219f115f0 RDI: 0000000000000000 [ 317.401654] RBP: ffff880219f11838 R08: 0000000000000000 R09: 0000000000000000 [ 317.401671] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880219f11300 [ 317.401689] R13: ffff880219f17770 R14: ffff88022c1daef8 R15: ffffffffa06ae950 [ 317.401707] FS: 00007febf77a9980(0000) GS:ffff880236d40000(0000) knlGS:0000000000000000 [ 317.401727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 317.401743] CR2: 0000000000000030 CR3: 0000000222072003 CR4: 00000000003606e0 [ 317.401761] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 317.401779] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 317.401796] Call Trace: [ 317.401894] guc_client_free+0x9/0x130 [i915] [ 317.401993] intel_guc_submission_fini+0x50/0x90 [i915] [ 317.402092] intel_uc_fini+0x34/0xd0 [i915] [ 317.402179] i915_gem_fini+0x5c/0x100 [i915] [ 317.402249] i915_driver_unload+0xd2/0x110 [i915] [ 317.402321] i915_pci_remove+0x10/0x20 [i915] [ 317.402341] pci_device_remove+0x36/0xb0 [ 317.402357] device_release_driver_internal+0x185/0x250 [ 317.402374] driver_detach+0x35/0x70 [ 317.402390] bus_remove_driver+0x53/0xd0 [ 317.402404] pci_unregister_driver+0x25/0xa0 [ 317.402423] __se_sys_delete_module+0x162/0x210 [ 317.402439] ? do_syscall_64+0xd/0x190 [ 317.402454] do_syscall_64+0x55/0x190 [ 317.402470] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 317.402485] RIP: 0033:0x7febf6e5d1b7 [ 317.402496] Code: 73 01 c3 48 8b 0d d1 8c 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a1 8c 2c 00 f7 d8 64 89 01 48 [ 317.402646] RSP: 002b:00007fffb5e72798 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 317.402667] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007febf6e5d1b7 [ 317.402686] RDX: 0000000000000000 RSI: 0000000000000800 RDI: 0000562da1addd98 [ 317.402703] RBP: 0000562da1addd30 R08: 0000562da1addd9c R09: 00007fffb5e727d8 [ 317.402721] R10: 00007fffb5e71794 R11: 0000000000000206 R12: 0000562da0ff6470 Testcase: igt/drv_module_reload/basic-reload-inject Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20180712202027.19801-1-chris@chris-wilson.co.uk
2018-07-13 03:20:27 +07:00
if (client)
guc_client_free(client);
}
static int __guc_client_enable(struct intel_guc_client *client)
{
int ret;
guc_proc_desc_init(client);
guc_stage_desc_init(client);
ret = create_doorbell(client);
if (ret)
goto fail;
return 0;
fail:
guc_stage_desc_fini(client);
guc_proc_desc_fini(client);
return ret;
}
static void __guc_client_disable(struct intel_guc_client *client)
{
/*
* By the time we're here, GuC may have already been reset. if that is
* the case, instead of trying (in vain) to communicate with it, let's
* just cleanup the doorbell HW and our internal state.
*/
if (intel_guc_is_alive(client->guc))
destroy_doorbell(client);
else
__fini_doorbell(client);
guc_stage_desc_fini(client);
guc_proc_desc_fini(client);
}
static int guc_clients_enable(struct intel_guc *guc)
{
int ret;
ret = __guc_client_enable(guc->execbuf_client);
if (ret)
return ret;
if (guc->preempt_client) {
ret = __guc_client_enable(guc->preempt_client);
if (ret) {
__guc_client_disable(guc->execbuf_client);
return ret;
}
}
return 0;
}
static void guc_clients_disable(struct intel_guc *guc)
{
if (guc->preempt_client)
__guc_client_disable(guc->preempt_client);
if (guc->execbuf_client)
__guc_client_disable(guc->execbuf_client);
}
/*
* Set up the memory resources to be shared with the GuC (via the GGTT)
* at firmware loading time.
*/
int intel_guc_submission_init(struct intel_guc *guc)
{
drm/i915/guc: Move GuC workqueue allocations outside of the mutex This gets rid of the following lockdep splat: ====================================================== WARNING: possible circular locking dependency detected 4.15.0-rc2-CI-Patchwork_7428+ #1 Not tainted ------------------------------------------------------ debugfs_test/1351 is trying to acquire lock: (&dev->struct_mutex){+.+.}, at: [<000000009d90d1a3>] i915_mutex_lock_interruptible+0x47/0x130 [i915] but task is already holding lock: (&mm->mmap_sem){++++}, at: [<000000005df01c1e>] __do_page_fault+0x106/0x560 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #6 (&mm->mmap_sem){++++}: __might_fault+0x63/0x90 _copy_to_user+0x1e/0x70 filldir+0x8c/0xf0 dcache_readdir+0xeb/0x160 iterate_dir+0xe6/0x150 SyS_getdents+0xa0/0x130 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #5 (&sb->s_type->i_mutex_key#5){++++}: lockref_get+0x9/0x20 -> #4 ((completion)&req.done){+.+.}: wait_for_common+0x54/0x210 devtmpfs_create_node+0x130/0x150 device_add+0x5ad/0x5e0 device_create_groups_vargs+0xd4/0xe0 device_create+0x35/0x40 msr_device_create+0x22/0x40 cpuhp_invoke_callback+0xc5/0xbf0 cpuhp_thread_fun+0x167/0x210 smpboot_thread_fn+0x17f/0x270 kthread+0x173/0x1b0 ret_from_fork+0x24/0x30 -> #3 (cpuhp_state-up){+.+.}: cpuhp_issue_call+0x132/0x1c0 __cpuhp_setup_state_cpuslocked+0x12f/0x2a0 __cpuhp_setup_state+0x3a/0x50 page_writeback_init+0x3a/0x5c start_kernel+0x393/0x3e2 secondary_startup_64+0xa5/0xb0 -> #2 (cpuhp_state_mutex){+.+.}: __mutex_lock+0x81/0x9b0 __cpuhp_setup_state_cpuslocked+0x4b/0x2a0 __cpuhp_setup_state+0x3a/0x50 page_alloc_init+0x1f/0x26 start_kernel+0x139/0x3e2 secondary_startup_64+0xa5/0xb0 -> #1 (cpu_hotplug_lock.rw_sem){++++}: cpus_read_lock+0x34/0xa0 apply_workqueue_attrs+0xd/0x40 __alloc_workqueue_key+0x2c7/0x4e1 intel_guc_submission_init+0x10c/0x650 [i915] intel_uc_init_hw+0x29e/0x460 [i915] i915_gem_init_hw+0xca/0x290 [i915] i915_gem_init+0x115/0x3a0 [i915] i915_driver_load+0x9a8/0x16c0 [i915] i915_pci_probe+0x2e/0x90 [i915] pci_device_probe+0x9c/0x120 driver_probe_device+0x2a3/0x480 __driver_attach+0xd9/0xe0 bus_for_each_dev+0x57/0x90 bus_add_driver+0x168/0x260 driver_register+0x52/0xc0 do_one_initcall+0x39/0x150 do_init_module+0x56/0x1ef load_module+0x231c/0x2d70 SyS_finit_module+0xa5/0xe0 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #0 (&dev->struct_mutex){+.+.}: lock_acquire+0xaf/0x200 __mutex_lock+0x81/0x9b0 i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_gem_fault+0x201/0x760 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x85b/0xe40 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x22/0x30 other info that might help us debug this: Chain exists of: &dev->struct_mutex --> &sb->s_type->i_mutex_key#5 --> &mm->mmap_sem Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&mm->mmap_sem); lock(&sb->s_type->i_mutex_key#5); lock(&mm->mmap_sem); lock(&dev->struct_mutex); *** DEADLOCK *** 1 lock held by debugfs_test/1351: #0: (&mm->mmap_sem){++++}, at: [<000000005df01c1e>] __do_page_fault+0x106/0x560 stack backtrace: CPU: 2 PID: 1351 Comm: debugfs_test Not tainted 4.15.0-rc2-CI-Patchwork_7428+ #1 Hardware name: /NUC6i5SYB, BIOS SYSKLi35.86A.0057.2017.0119.1758 01/19/2017 Call Trace: dump_stack+0x5f/0x86 print_circular_bug+0x230/0x3b0 check_prev_add+0x439/0x7b0 ? lockdep_init_map_crosslock+0x20/0x20 ? unwind_get_return_address+0x16/0x30 ? __lock_acquire+0x1385/0x15a0 __lock_acquire+0x1385/0x15a0 lock_acquire+0xaf/0x200 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] __mutex_lock+0x81/0x9b0 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_mutex_lock_interruptible+0x47/0x130 [i915] ? __pm_runtime_resume+0x4f/0x80 i915_gem_fault+0x201/0x760 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x85b/0xe40 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x22/0x30 RIP: 0033:0x7f98d6f49116 RSP: 002b:00007ffd6ffc3278 EFLAGS: 00010283 RAX: 00007f98d39a2bc0 RBX: 0000000000000000 RCX: 0000000000001680 RDX: 0000000000001680 RSI: 00007ffd6ffc3400 RDI: 00007f98d39a2bc0 RBP: 00007ffd6ffc33a0 R08: 0000000000000000 R09: 00000000000005a0 R10: 000055e847c2a830 R11: 0000000000000002 R12: 0000000000000001 R13: 000055e847c1d040 R14: 00007ffd6ffc3400 R15: 00007f98d6752ba0 v2: Init preempt_work unconditionally (Chris) v3: Mention that we need the enable_guc=1 for lockdep splat (Chris) Testcase: igt/debugfs_test/read_all_entries # with i915.enable_guc=1 Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171213221352.7173-2-michal.winiarski@intel.com
2017-12-14 05:13:47 +07:00
struct drm_i915_private *dev_priv = guc_to_i915(guc);
struct intel_engine_cs *engine;
enum intel_engine_id id;
int ret;
if (guc->stage_desc_pool)
return 0;
ret = guc_stage_desc_pool_create(guc);
if (ret)
return ret;
/*
* Keep static analysers happy, let them know that we allocated the
* vma after testing that it didn't exist earlier.
*/
GEM_BUG_ON(!guc->stage_desc_pool);
WARN_ON(!guc_verify_doorbells(guc));
ret = guc_clients_create(guc);
if (ret)
goto err_pool;
drm/i915/guc: Move GuC workqueue allocations outside of the mutex This gets rid of the following lockdep splat: ====================================================== WARNING: possible circular locking dependency detected 4.15.0-rc2-CI-Patchwork_7428+ #1 Not tainted ------------------------------------------------------ debugfs_test/1351 is trying to acquire lock: (&dev->struct_mutex){+.+.}, at: [<000000009d90d1a3>] i915_mutex_lock_interruptible+0x47/0x130 [i915] but task is already holding lock: (&mm->mmap_sem){++++}, at: [<000000005df01c1e>] __do_page_fault+0x106/0x560 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #6 (&mm->mmap_sem){++++}: __might_fault+0x63/0x90 _copy_to_user+0x1e/0x70 filldir+0x8c/0xf0 dcache_readdir+0xeb/0x160 iterate_dir+0xe6/0x150 SyS_getdents+0xa0/0x130 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #5 (&sb->s_type->i_mutex_key#5){++++}: lockref_get+0x9/0x20 -> #4 ((completion)&req.done){+.+.}: wait_for_common+0x54/0x210 devtmpfs_create_node+0x130/0x150 device_add+0x5ad/0x5e0 device_create_groups_vargs+0xd4/0xe0 device_create+0x35/0x40 msr_device_create+0x22/0x40 cpuhp_invoke_callback+0xc5/0xbf0 cpuhp_thread_fun+0x167/0x210 smpboot_thread_fn+0x17f/0x270 kthread+0x173/0x1b0 ret_from_fork+0x24/0x30 -> #3 (cpuhp_state-up){+.+.}: cpuhp_issue_call+0x132/0x1c0 __cpuhp_setup_state_cpuslocked+0x12f/0x2a0 __cpuhp_setup_state+0x3a/0x50 page_writeback_init+0x3a/0x5c start_kernel+0x393/0x3e2 secondary_startup_64+0xa5/0xb0 -> #2 (cpuhp_state_mutex){+.+.}: __mutex_lock+0x81/0x9b0 __cpuhp_setup_state_cpuslocked+0x4b/0x2a0 __cpuhp_setup_state+0x3a/0x50 page_alloc_init+0x1f/0x26 start_kernel+0x139/0x3e2 secondary_startup_64+0xa5/0xb0 -> #1 (cpu_hotplug_lock.rw_sem){++++}: cpus_read_lock+0x34/0xa0 apply_workqueue_attrs+0xd/0x40 __alloc_workqueue_key+0x2c7/0x4e1 intel_guc_submission_init+0x10c/0x650 [i915] intel_uc_init_hw+0x29e/0x460 [i915] i915_gem_init_hw+0xca/0x290 [i915] i915_gem_init+0x115/0x3a0 [i915] i915_driver_load+0x9a8/0x16c0 [i915] i915_pci_probe+0x2e/0x90 [i915] pci_device_probe+0x9c/0x120 driver_probe_device+0x2a3/0x480 __driver_attach+0xd9/0xe0 bus_for_each_dev+0x57/0x90 bus_add_driver+0x168/0x260 driver_register+0x52/0xc0 do_one_initcall+0x39/0x150 do_init_module+0x56/0x1ef load_module+0x231c/0x2d70 SyS_finit_module+0xa5/0xe0 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #0 (&dev->struct_mutex){+.+.}: lock_acquire+0xaf/0x200 __mutex_lock+0x81/0x9b0 i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_gem_fault+0x201/0x760 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x85b/0xe40 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x22/0x30 other info that might help us debug this: Chain exists of: &dev->struct_mutex --> &sb->s_type->i_mutex_key#5 --> &mm->mmap_sem Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&mm->mmap_sem); lock(&sb->s_type->i_mutex_key#5); lock(&mm->mmap_sem); lock(&dev->struct_mutex); *** DEADLOCK *** 1 lock held by debugfs_test/1351: #0: (&mm->mmap_sem){++++}, at: [<000000005df01c1e>] __do_page_fault+0x106/0x560 stack backtrace: CPU: 2 PID: 1351 Comm: debugfs_test Not tainted 4.15.0-rc2-CI-Patchwork_7428+ #1 Hardware name: /NUC6i5SYB, BIOS SYSKLi35.86A.0057.2017.0119.1758 01/19/2017 Call Trace: dump_stack+0x5f/0x86 print_circular_bug+0x230/0x3b0 check_prev_add+0x439/0x7b0 ? lockdep_init_map_crosslock+0x20/0x20 ? unwind_get_return_address+0x16/0x30 ? __lock_acquire+0x1385/0x15a0 __lock_acquire+0x1385/0x15a0 lock_acquire+0xaf/0x200 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] __mutex_lock+0x81/0x9b0 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_mutex_lock_interruptible+0x47/0x130 [i915] ? __pm_runtime_resume+0x4f/0x80 i915_gem_fault+0x201/0x760 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x85b/0xe40 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x22/0x30 RIP: 0033:0x7f98d6f49116 RSP: 002b:00007ffd6ffc3278 EFLAGS: 00010283 RAX: 00007f98d39a2bc0 RBX: 0000000000000000 RCX: 0000000000001680 RDX: 0000000000001680 RSI: 00007ffd6ffc3400 RDI: 00007f98d39a2bc0 RBP: 00007ffd6ffc33a0 R08: 0000000000000000 R09: 00000000000005a0 R10: 000055e847c2a830 R11: 0000000000000002 R12: 0000000000000001 R13: 000055e847c1d040 R14: 00007ffd6ffc3400 R15: 00007f98d6752ba0 v2: Init preempt_work unconditionally (Chris) v3: Mention that we need the enable_guc=1 for lockdep splat (Chris) Testcase: igt/debugfs_test/read_all_entries # with i915.enable_guc=1 Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171213221352.7173-2-michal.winiarski@intel.com
2017-12-14 05:13:47 +07:00
for_each_engine(engine, dev_priv, id) {
guc->preempt_work[id].engine = engine;
INIT_WORK(&guc->preempt_work[id].work, inject_preempt_context);
}
return 0;
err_pool:
guc_stage_desc_pool_destroy(guc);
return ret;
}
void intel_guc_submission_fini(struct intel_guc *guc)
{
drm/i915/guc: Move GuC workqueue allocations outside of the mutex This gets rid of the following lockdep splat: ====================================================== WARNING: possible circular locking dependency detected 4.15.0-rc2-CI-Patchwork_7428+ #1 Not tainted ------------------------------------------------------ debugfs_test/1351 is trying to acquire lock: (&dev->struct_mutex){+.+.}, at: [<000000009d90d1a3>] i915_mutex_lock_interruptible+0x47/0x130 [i915] but task is already holding lock: (&mm->mmap_sem){++++}, at: [<000000005df01c1e>] __do_page_fault+0x106/0x560 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #6 (&mm->mmap_sem){++++}: __might_fault+0x63/0x90 _copy_to_user+0x1e/0x70 filldir+0x8c/0xf0 dcache_readdir+0xeb/0x160 iterate_dir+0xe6/0x150 SyS_getdents+0xa0/0x130 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #5 (&sb->s_type->i_mutex_key#5){++++}: lockref_get+0x9/0x20 -> #4 ((completion)&req.done){+.+.}: wait_for_common+0x54/0x210 devtmpfs_create_node+0x130/0x150 device_add+0x5ad/0x5e0 device_create_groups_vargs+0xd4/0xe0 device_create+0x35/0x40 msr_device_create+0x22/0x40 cpuhp_invoke_callback+0xc5/0xbf0 cpuhp_thread_fun+0x167/0x210 smpboot_thread_fn+0x17f/0x270 kthread+0x173/0x1b0 ret_from_fork+0x24/0x30 -> #3 (cpuhp_state-up){+.+.}: cpuhp_issue_call+0x132/0x1c0 __cpuhp_setup_state_cpuslocked+0x12f/0x2a0 __cpuhp_setup_state+0x3a/0x50 page_writeback_init+0x3a/0x5c start_kernel+0x393/0x3e2 secondary_startup_64+0xa5/0xb0 -> #2 (cpuhp_state_mutex){+.+.}: __mutex_lock+0x81/0x9b0 __cpuhp_setup_state_cpuslocked+0x4b/0x2a0 __cpuhp_setup_state+0x3a/0x50 page_alloc_init+0x1f/0x26 start_kernel+0x139/0x3e2 secondary_startup_64+0xa5/0xb0 -> #1 (cpu_hotplug_lock.rw_sem){++++}: cpus_read_lock+0x34/0xa0 apply_workqueue_attrs+0xd/0x40 __alloc_workqueue_key+0x2c7/0x4e1 intel_guc_submission_init+0x10c/0x650 [i915] intel_uc_init_hw+0x29e/0x460 [i915] i915_gem_init_hw+0xca/0x290 [i915] i915_gem_init+0x115/0x3a0 [i915] i915_driver_load+0x9a8/0x16c0 [i915] i915_pci_probe+0x2e/0x90 [i915] pci_device_probe+0x9c/0x120 driver_probe_device+0x2a3/0x480 __driver_attach+0xd9/0xe0 bus_for_each_dev+0x57/0x90 bus_add_driver+0x168/0x260 driver_register+0x52/0xc0 do_one_initcall+0x39/0x150 do_init_module+0x56/0x1ef load_module+0x231c/0x2d70 SyS_finit_module+0xa5/0xe0 entry_SYSCALL_64_fastpath+0x1c/0x89 -> #0 (&dev->struct_mutex){+.+.}: lock_acquire+0xaf/0x200 __mutex_lock+0x81/0x9b0 i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_gem_fault+0x201/0x760 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x85b/0xe40 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x22/0x30 other info that might help us debug this: Chain exists of: &dev->struct_mutex --> &sb->s_type->i_mutex_key#5 --> &mm->mmap_sem Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&mm->mmap_sem); lock(&sb->s_type->i_mutex_key#5); lock(&mm->mmap_sem); lock(&dev->struct_mutex); *** DEADLOCK *** 1 lock held by debugfs_test/1351: #0: (&mm->mmap_sem){++++}, at: [<000000005df01c1e>] __do_page_fault+0x106/0x560 stack backtrace: CPU: 2 PID: 1351 Comm: debugfs_test Not tainted 4.15.0-rc2-CI-Patchwork_7428+ #1 Hardware name: /NUC6i5SYB, BIOS SYSKLi35.86A.0057.2017.0119.1758 01/19/2017 Call Trace: dump_stack+0x5f/0x86 print_circular_bug+0x230/0x3b0 check_prev_add+0x439/0x7b0 ? lockdep_init_map_crosslock+0x20/0x20 ? unwind_get_return_address+0x16/0x30 ? __lock_acquire+0x1385/0x15a0 __lock_acquire+0x1385/0x15a0 lock_acquire+0xaf/0x200 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] __mutex_lock+0x81/0x9b0 ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] ? i915_mutex_lock_interruptible+0x47/0x130 [i915] i915_mutex_lock_interruptible+0x47/0x130 [i915] ? __pm_runtime_resume+0x4f/0x80 i915_gem_fault+0x201/0x760 [i915] __do_fault+0x15/0x70 __handle_mm_fault+0x85b/0xe40 handle_mm_fault+0x14f/0x2f0 __do_page_fault+0x2d1/0x560 page_fault+0x22/0x30 RIP: 0033:0x7f98d6f49116 RSP: 002b:00007ffd6ffc3278 EFLAGS: 00010283 RAX: 00007f98d39a2bc0 RBX: 0000000000000000 RCX: 0000000000001680 RDX: 0000000000001680 RSI: 00007ffd6ffc3400 RDI: 00007f98d39a2bc0 RBP: 00007ffd6ffc33a0 R08: 0000000000000000 R09: 00000000000005a0 R10: 000055e847c2a830 R11: 0000000000000002 R12: 0000000000000001 R13: 000055e847c1d040 R14: 00007ffd6ffc3400 R15: 00007f98d6752ba0 v2: Init preempt_work unconditionally (Chris) v3: Mention that we need the enable_guc=1 for lockdep splat (Chris) Testcase: igt/debugfs_test/read_all_entries # with i915.enable_guc=1 Signed-off-by: Michał Winiarski <michal.winiarski@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Link: https://patchwork.freedesktop.org/patch/msgid/20171213221352.7173-2-michal.winiarski@intel.com
2017-12-14 05:13:47 +07:00
struct drm_i915_private *dev_priv = guc_to_i915(guc);
struct intel_engine_cs *engine;
enum intel_engine_id id;
for_each_engine(engine, dev_priv, id)
cancel_work_sync(&guc->preempt_work[id].work);
guc_clients_destroy(guc);
WARN_ON(!guc_verify_doorbells(guc));
drm/i915/guc: Protect against no desc-pool on premature shutdown Hopefully the final hack to get guc fault-injection happy before we can clean it up again, starting from a known good baseline... [ 383.017530] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0 [ 383.017556] Oops: 0000 [#1] PREEMPT SMP PTI [ 383.017566] CPU: 7 PID: 4725 Comm: drv_module_relo Tainted: G U 4.18.0-rc4-CI-CI_DRM_4485+ #1 [ 383.017581] Hardware name: Micro-Star International Co., Ltd. MS-7B54/Z370M MORTAR (MS-7B54), BIOS 1.10 12/28/2017 [ 383.017664] RIP: 0010:guc_stage_desc_pool_destroy+0x17/0xe0 [i915] [ 383.017674] Code: 59 a0 c6 05 02 59 18 00 01 e8 5e 01 c3 e0 eb b1 0f 1f 00 53 48 89 fb 48 81 c7 90 02 00 00 e8 60 64 45 e1 48 8b 83 80 02 00 00 <48> 8b 80 a0 00 00 00 48 8b 90 68 02 00 00 48 83 ea 01 48 81 fa ff [ 383.017771] RSP: 0018:ffffc900004bbdd0 EFLAGS: 00010282 [ 383.017782] RAX: 0000000000000000 RBX: ffff88012ff41300 RCX: 0000000000000000 [ 383.017794] RDX: 0000000000000000 RSI: ffffc900004bbd80 RDI: 0000000000000000 [ 383.017805] RBP: ffff88012ff40000 R08: 00000000d876ee11 R09: 0000000000000000 [ 383.017817] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88012ff47770 [ 383.017828] R13: ffff88012ff40068 R14: ffff880264392ef8 R15: ffffffffa0639950 [ 383.017840] FS: 00007fb9c18c8980(0000) GS:ffff8802663c0000(0000) knlGS:0000000000000000 [ 383.017853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 383.017864] CR2: 00000000000000a0 CR3: 00000001df6cc003 CR4: 00000000003606e0 [ 383.017875] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 383.017887] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 383.017898] Call Trace: [ 383.017962] intel_uc_fini+0x34/0xd0 [i915] [ 383.018020] i915_gem_fini+0x5c/0x100 [i915] [ 383.018093] i915_driver_unload+0xd2/0x110 [i915] [ 383.018150] i915_pci_remove+0x10/0x20 [i915] [ 383.018165] pci_device_remove+0x36/0xb0 [ 383.018179] device_release_driver_internal+0x185/0x250 [ 383.018193] driver_detach+0x35/0x70 [ 383.018205] bus_remove_driver+0x53/0xd0 [ 383.018217] pci_unregister_driver+0x25/0xa0 [ 383.018232] __se_sys_delete_module+0x162/0x210 [ 383.018245] ? do_syscall_64+0xd/0x190 [ 383.018257] do_syscall_64+0x55/0x190 [ 383.018270] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 383.018282] RIP: 0033:0x7fb9c0f7c1b7 [ 383.018290] Code: 73 01 c3 48 8b 0d d1 8c 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a1 8c 2c 00 f7 d8 64 89 01 48 [ 383.018408] RSP: 002b:00007fffa01c2aa8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 383.018425] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb9c0f7c1b7 [ 383.018440] RDX: 0000000000000000 RSI: 0000000000000800 RDI: 0000560b96856d48 [ 383.018454] RBP: 0000560b96856ce0 R08: 0000560b96856d4c R09: 00007fffa01c2ae8 [ 383.018468] R10: 00007fffa01c1aa4 R11: 0000000000000206 R12: 0000560b954f7470 Testcase: igt/drv_module_reload/basic-reload-inject Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Michał Winiarski <michal.winiarski@intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com> Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20180713172658.14070-1-chris@chris-wilson.co.uk
2018-07-14 00:26:58 +07:00
if (guc->stage_desc_pool)
guc_stage_desc_pool_destroy(guc);
}
static void guc_interrupts_capture(struct drm_i915_private *dev_priv)
{
struct intel_rps *rps = &dev_priv->gt_pm.rps;
struct intel_engine_cs *engine;
enum intel_engine_id id;
int irqs;
/* tell all command streamers to forward interrupts (but not vblank)
* to GuC
*/
irqs = _MASKED_BIT_ENABLE(GFX_INTERRUPT_STEERING);
for_each_engine(engine, dev_priv, id)
I915_WRITE(RING_MODE_GEN7(engine), irqs);
/* route USER_INTERRUPT to Host, all others are sent to GuC. */
irqs = GT_RENDER_USER_INTERRUPT << GEN8_RCS_IRQ_SHIFT |
GT_RENDER_USER_INTERRUPT << GEN8_BCS_IRQ_SHIFT;
/* These three registers have the same bit definitions */
I915_WRITE(GUC_BCS_RCS_IER, ~irqs);
I915_WRITE(GUC_VCS2_VCS1_IER, ~irqs);
I915_WRITE(GUC_WD_VECS_IER, ~irqs);
/*
* The REDIRECT_TO_GUC bit of the PMINTRMSK register directs all
* (unmasked) PM interrupts to the GuC. All other bits of this
* register *disable* generation of a specific interrupt.
*
* 'pm_intrmsk_mbz' indicates bits that are NOT to be set when
* writing to the PM interrupt mask register, i.e. interrupts
* that must not be disabled.
*
* If the GuC is handling these interrupts, then we must not let
* the PM code disable ANY interrupt that the GuC is expecting.
* So for each ENABLED (0) bit in this register, we must SET the
* bit in pm_intrmsk_mbz so that it's left enabled for the GuC.
* GuC needs ARAT expired interrupt unmasked hence it is set in
* pm_intrmsk_mbz.
*
* Here we CLEAR REDIRECT_TO_GUC bit in pm_intrmsk_mbz, which will
* result in the register bit being left SET!
*/
rps->pm_intrmsk_mbz |= ARAT_EXPIRED_INTRMSK;
rps->pm_intrmsk_mbz &= ~GEN8_PMINTR_DISABLE_REDIRECT_TO_GUC;
}
static void guc_interrupts_release(struct drm_i915_private *dev_priv)
{
struct intel_rps *rps = &dev_priv->gt_pm.rps;
struct intel_engine_cs *engine;
enum intel_engine_id id;
int irqs;
/*
* tell all command streamers NOT to forward interrupts or vblank
* to GuC.
*/
irqs = _MASKED_FIELD(GFX_FORWARD_VBLANK_MASK, GFX_FORWARD_VBLANK_NEVER);
irqs |= _MASKED_BIT_DISABLE(GFX_INTERRUPT_STEERING);
for_each_engine(engine, dev_priv, id)
I915_WRITE(RING_MODE_GEN7(engine), irqs);
/* route all GT interrupts to the host */
I915_WRITE(GUC_BCS_RCS_IER, 0);
I915_WRITE(GUC_VCS2_VCS1_IER, 0);
I915_WRITE(GUC_WD_VECS_IER, 0);
rps->pm_intrmsk_mbz |= GEN8_PMINTR_DISABLE_REDIRECT_TO_GUC;
rps->pm_intrmsk_mbz &= ~ARAT_EXPIRED_INTRMSK;
}
static void guc_submission_park(struct intel_engine_cs *engine)
drm/i915/guc: Always enable the breadcrumbs irq The execlists emulation on top of the GuC (used for scheduling and preemption) depends on the MI_USER_INTERRUPT for its notifications and tasklet action. As we always employ the irq, there is no advantage in ever disabling it while we are using the GuC, so allow us to arm the breadcrumb irq when enabling GuC submission and disarm upon disabling. The impact should be lessened by the delayed irq disabling we do (we only disable after receiving an interrupt for which no one was wanting), but allowing guc to explicitly manage the irq in relation to itself is simpler and prevents an issue with losing an interrupt for preemption as it is not coupled to an active request. Internally, we add a reference counter (breadcrumbs.irq_enabled) as a simple mechanism to allow GuC to keep the breadcrumb irq enabled. To improve upon always enabling the irq while guc is selected, we need to hook into the parking facility of intel_engines so that we only enable the breadcrumbs while the GT is active (one step better would be to individually park/unpark each engine). In effect, this means that we keep the breadcrumb irq always enabled for the entire duration the guc is busy, whereas before we would try to switch it off whenever we idled for more than interrupt with no associated waiters. The difference *should* be negligible in practice! v2: Stop abusing fence signaling (and its auxiliary data structures) to enable the breadcrumbs irqs. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michał Winiarski <michal.winiarski@intel.com>, Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>, Link: https://patchwork.freedesktop.org/patch/msgid/20171025143943.7661-3-chris@chris-wilson.co.uk
2017-10-25 21:39:42 +07:00
{
intel_engine_unpin_breadcrumbs_irq(engine);
}
static void guc_submission_unpark(struct intel_engine_cs *engine)
drm/i915/guc: Always enable the breadcrumbs irq The execlists emulation on top of the GuC (used for scheduling and preemption) depends on the MI_USER_INTERRUPT for its notifications and tasklet action. As we always employ the irq, there is no advantage in ever disabling it while we are using the GuC, so allow us to arm the breadcrumb irq when enabling GuC submission and disarm upon disabling. The impact should be lessened by the delayed irq disabling we do (we only disable after receiving an interrupt for which no one was wanting), but allowing guc to explicitly manage the irq in relation to itself is simpler and prevents an issue with losing an interrupt for preemption as it is not coupled to an active request. Internally, we add a reference counter (breadcrumbs.irq_enabled) as a simple mechanism to allow GuC to keep the breadcrumb irq enabled. To improve upon always enabling the irq while guc is selected, we need to hook into the parking facility of intel_engines so that we only enable the breadcrumbs while the GT is active (one step better would be to individually park/unpark each engine). In effect, this means that we keep the breadcrumb irq always enabled for the entire duration the guc is busy, whereas before we would try to switch it off whenever we idled for more than interrupt with no associated waiters. The difference *should* be negligible in practice! v2: Stop abusing fence signaling (and its auxiliary data structures) to enable the breadcrumbs irqs. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michał Winiarski <michal.winiarski@intel.com>, Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>, Link: https://patchwork.freedesktop.org/patch/msgid/20171025143943.7661-3-chris@chris-wilson.co.uk
2017-10-25 21:39:42 +07:00
{
intel_engine_pin_breadcrumbs_irq(engine);
}
static void guc_set_default_submission(struct intel_engine_cs *engine)
{
/*
* We inherit a bunch of functions from execlists that we'd like
* to keep using:
*
* engine->submit_request = execlists_submit_request;
* engine->cancel_requests = execlists_cancel_requests;
* engine->schedule = execlists_schedule;
*
* But we need to override the actual submission backend in order
* to talk to the GuC.
*/
intel_execlists_set_default_submission(engine);
engine->execlists.tasklet.func = guc_submission_tasklet;
engine->park = guc_submission_park;
engine->unpark = guc_submission_unpark;
engine->reset.prepare = guc_reset_prepare;
engine->flags &= ~I915_ENGINE_SUPPORTS_STATS;
}
int intel_guc_submission_enable(struct intel_guc *guc)
{
struct drm_i915_private *dev_priv = guc_to_i915(guc);
struct intel_engine_cs *engine;
drm/i915: Allocate intel_engine_cs structure only for the enabled engines With the possibility of addition of many more number of rings in future, the drm_i915_private structure could bloat as an array, of type intel_engine_cs, is embedded inside it. struct intel_engine_cs engine[I915_NUM_ENGINES]; Though this is still fine as generally there is only a single instance of drm_i915_private structure used, but not all of the possible rings would be enabled or active on most of the platforms. Some memory can be saved by allocating intel_engine_cs structure only for the enabled/active engines. Currently the engine/ring ID is kept static and dev_priv->engine[] is simply indexed using the enums defined in intel_engine_id. To save memory and continue using the static engine/ring IDs, 'engine' is defined as an array of pointers. struct intel_engine_cs *engine[I915_NUM_ENGINES]; dev_priv->engine[engine_ID] will be NULL for disabled engine instances. There is a text size reduction of 928 bytes, from 1028200 to 1027272, for i915.o file (but for i915.ko file text size remain same as 1193131 bytes). v2: - Remove the engine iterator field added in drm_i915_private structure, instead pass a local iterator variable to the for_each_engine** macros. (Chris) - Do away with intel_engine_initialized() and instead directly use the NULL pointer check on engine pointer. (Chris) v3: - Remove for_each_engine_id() macro, as the updated macro for_each_engine() can be used in place of it. (Chris) - Protect the access to Render engine Fault register with a NULL check, as engine specific init is done later in Driver load sequence. v4: - Use !!dev_priv->engine[VCS] style for the engine check in getparam. (Chris) - Kill the superfluous init_engine_lists(). v5: - Cleanup the intel_engines_init() & intel_engines_setup(), with respect to allocation of intel_engine_cs structure. (Chris) v6: - Rebase. v7: - Optimize the for_each_engine_masked() macro. (Chris) - Change the type of 'iter' local variable to enum intel_engine_id. (Chris) - Rebase. v8: Rebase. v9: Rebase. v10: - For index calculation use engine ID instead of pointer based arithmetic in intel_engine_sync_index() as engine pointers are not contiguous now (Chris) - For appropriateness, rename local enum variable 'iter' to 'id'. (Joonas) - Use for_each_engine macro for cleanup in intel_engines_init() and remove check for NULL engine pointer in cleanup() routines. (Joonas) v11: Rebase. Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Akash Goel <akash.goel@intel.com> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/1476378888-7372-1-git-send-email-akash.goel@intel.com
2016-10-14 00:14:48 +07:00
enum intel_engine_id id;
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
int err;
/*
* We're using GuC work items for submitting work through GuC. Since
* we're coalescing multiple requests from a single context into a
* single work item prior to assigning it to execlist_port, we can
* never have more work items than the total number of ports (for all
* engines). The GuC firmware is controlling the HEAD of work queue,
* and it is guaranteed that it will remove the work item from the
* queue before our request is completed.
*/
BUILD_BUG_ON(ARRAY_SIZE(engine->execlists.port) *
sizeof(struct guc_wq_item) *
I915_NUM_ENGINES > GUC_WQ_SIZE);
GEM_BUG_ON(!guc->execbuf_client);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
err = intel_guc_sample_forcewake(guc);
if (err)
return err;
err = guc_clients_enable(guc);
drm/i915/guc: Sanitize GuC client initialization Started adding proper teardown to guc_client_alloc, ended up removing quite a few dead ends where errors communicating with the GuC were silently ignored. There also seemed to be quite a few erronous teardown actions performed in case of an error (ordering wrong). v2: - Increase function symmetry/proximity (Michal/Daniele) - Fix __reserve_doorbell accounting for high priority (Daniele) - Call __update_doorbell_desc! (Daniele) - Isolate __guc_{,de}allocate_doorbell (Michal/Daniele) v3: - "Select" a cacheline is a more accurate verb than "reserve" (Daniele). - We cannot update & create the doorbell without reserving it first, so move the whole doorbell creation for execbuf_client to the submission enable (Oscar).i - Add a fixme for ignoring possible doorbell destroy errors. v4: - Remove comment about is_high_priority (Daniele) - Debug message typo (Daniele) - Reuse __get_doorbell in more places (Daniele) - Do not do arithmetic on void pointers (Daniele) - Add comment to __reset_doorbell (Daniele) v5: - gccisms like arithmetic on void pointers are not frowned upon (Oscar) Signed-off-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Chris Wilson <chris@chris-wilson.co.uk> Signed-off-by: Oscar Mateo <oscar.mateo@intel.com> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
2017-03-23 00:39:44 +07:00
if (err)
return err;
/* Take over from manual control of ELSP (execlists) */
guc_interrupts_capture(dev_priv);
for_each_engine(engine, dev_priv, id) {
engine->set_default_submission = guc_set_default_submission;
engine->set_default_submission(engine);
drm/i915: Update reset path to fix incomplete requests Update reset path in preparation for engine reset which requires identification of incomplete requests and associated context and fixing their state so that engine can resume correctly after reset. The request that caused the hang will be skipped and head is reset to the start of breadcrumb. This allows us to resume from where we left-off. Since this request didn't complete normally we also need to cleanup elsp queue manually. This is vital if we employ nonblocking request submission where we may have a web of dependencies upon the hung request and so advancing the seqno manually is no longer trivial. ABI: gem_reset_stats / DRM_IOCTL_I915_GET_RESET_STATS We change the way we count pending batches. Only the active context involved in the reset is marked as either innocent or guilty, and not mark the entire world as pending. By inspection this only affects igt/gem_reset_stats (which assumes implementation details) and not piglit. ARB_robustness gives this guide on how we expect the user of this interface to behave: * Provide a mechanism for an OpenGL application to learn about graphics resets that affect the context. When a graphics reset occurs, the OpenGL context becomes unusable and the application must create a new context to continue operation. Detecting a graphics reset happens through an inexpensive query. And with regards to the actual meaning of the reset values: Certain events can result in a reset of the GL context. Such a reset causes all context state to be lost. Recovery from such events requires recreation of all objects in the affected context. The current status of the graphics reset state is returned by enum GetGraphicsResetStatusARB(); The symbolic constant returned indicates if the GL context has been in a reset state at any point since the last call to GetGraphicsResetStatusARB. NO_ERROR indicates that the GL context has not been in a reset state since the last call. GUILTY_CONTEXT_RESET_ARB indicates that a reset has been detected that is attributable to the current GL context. INNOCENT_CONTEXT_RESET_ARB indicates a reset has been detected that is not attributable to the current GL context. UNKNOWN_CONTEXT_RESET_ARB indicates a detected graphics reset whose cause is unknown. The language here is explicit in that we must mark up the guilty batch, but is loose enough for us to relax the innocent (i.e. pending) accounting as only the active batches are involved with the reset. In the future, we are looking towards single engine resetting (with minimal locking), where it seems inappropriate to mark the entire world as innocent since the reset occurred on a different engine. Reducing the information available means we only have to encounter the pain once, and also reduces the information leaking from one context to another. v2: Legacy ringbuffer submission required a reset following hibernation, or else we restore stale values to the RING_HEAD and walked over stolen garbage. v3: GuC requires replaying the requests after a reset. v4: Restore engine IRQ after reset (so waiters will be woken!) Rearm hangcheck if resetting with a waiter. Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Mika Kuoppala <mika.kuoppala@intel.com> Cc: Arun Siluvery <arun.siluvery@linux.intel.com> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Reviewed-by: Mika Kuoppala <mika.kuoppala@intel.com> Link: http://patchwork.freedesktop.org/patch/msgid/20160909131201.16673-13-chris@chris-wilson.co.uk
2016-09-09 20:11:53 +07:00
}
return 0;
}
void intel_guc_submission_disable(struct intel_guc *guc)
{
struct drm_i915_private *dev_priv = guc_to_i915(guc);
drm/i915/guc: Always enable the breadcrumbs irq The execlists emulation on top of the GuC (used for scheduling and preemption) depends on the MI_USER_INTERRUPT for its notifications and tasklet action. As we always employ the irq, there is no advantage in ever disabling it while we are using the GuC, so allow us to arm the breadcrumb irq when enabling GuC submission and disarm upon disabling. The impact should be lessened by the delayed irq disabling we do (we only disable after receiving an interrupt for which no one was wanting), but allowing guc to explicitly manage the irq in relation to itself is simpler and prevents an issue with losing an interrupt for preemption as it is not coupled to an active request. Internally, we add a reference counter (breadcrumbs.irq_enabled) as a simple mechanism to allow GuC to keep the breadcrumb irq enabled. To improve upon always enabling the irq while guc is selected, we need to hook into the parking facility of intel_engines so that we only enable the breadcrumbs while the GT is active (one step better would be to individually park/unpark each engine). In effect, this means that we keep the breadcrumb irq always enabled for the entire duration the guc is busy, whereas before we would try to switch it off whenever we idled for more than interrupt with no associated waiters. The difference *should* be negligible in practice! v2: Stop abusing fence signaling (and its auxiliary data structures) to enable the breadcrumbs irqs. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Cc: Michał Winiarski <michal.winiarski@intel.com>, Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com> Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>, Link: https://patchwork.freedesktop.org/patch/msgid/20171025143943.7661-3-chris@chris-wilson.co.uk
2017-10-25 21:39:42 +07:00
GEM_BUG_ON(dev_priv->gt.awake); /* GT should be parked first */
guc_interrupts_release(dev_priv);
guc_clients_disable(guc);
}
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/intel_guc.c"
#endif