2016-07-20 15:21:08 +07:00
|
|
|
/*
|
2018-02-21 16:56:36 +07:00
|
|
|
* Copyright © 2008-2018 Intel Corporation
|
2016-07-20 15:21:08 +07:00
|
|
|
*
|
|
|
|
* Permission is hereby granted, free of charge, to any person obtaining a
|
|
|
|
* copy of this software and associated documentation files (the "Software"),
|
|
|
|
* to deal in the Software without restriction, including without limitation
|
|
|
|
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
|
|
|
|
* and/or sell copies of the Software, and to permit persons to whom the
|
|
|
|
* Software is furnished to do so, subject to the following conditions:
|
|
|
|
*
|
|
|
|
* The above copyright notice and this permission notice (including the next
|
|
|
|
* paragraph) shall be included in all copies or substantial portions of the
|
|
|
|
* Software.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
|
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
|
|
|
|
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
|
|
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
|
|
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
|
|
|
|
* IN THE SOFTWARE.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
#ifndef I915_REQUEST_H
|
|
|
|
#define I915_REQUEST_H
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
#include <linux/dma-fence.h>
|
2019-04-03 15:21:32 +07:00
|
|
|
#include <linux/lockdep.h>
|
2016-07-20 15:21:11 +07:00
|
|
|
|
drm/i915/execlists: Preempt-to-busy
When using a global seqno, we required a precise stop-the-workd event to
handle preemption and unwind the global seqno counter. To accomplish
this, we would preempt to a special out-of-band context and wait for the
machine to report that it was idle. Given an idle machine, we could very
precisely see which requests had completed and which we needed to feed
back into the run queue.
However, now that we have scrapped the global seqno, we no longer need
to precisely unwind the global counter and only track requests by their
per-context seqno. This allows us to loosely unwind inflight requests
while scheduling a preemption, with the enormous caveat that the
requests we put back on the run queue are still _inflight_ (until the
preemption request is complete). This makes request tracking much more
messy, as at any point then we can see a completed request that we
believe is not currently scheduled for execution. We also have to be
careful not to rewind RING_TAIL past RING_HEAD on preempting to the
running context, and for this we use a semaphore to prevent completion
of the request before continuing.
To accomplish this feat, we change how we track requests scheduled to
the HW. Instead of appending our requests onto a single list as we
submit, we track each submission to ELSP as its own block. Then upon
receiving the CS preemption event, we promote the pending block to the
inflight block (discarding what was previously being tracked). As normal
CS completion events arrive, we then remove stale entries from the
inflight tracker.
v2: Be a tinge paranoid and ensure we flush the write into the HWS page
for the GPU semaphore to pick in a timely fashion.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190620142052.19311-1-chris@chris-wilson.co.uk
2019-06-20 21:20:51 +07:00
|
|
|
#include "gt/intel_context_types.h"
|
2019-05-22 04:11:31 +07:00
|
|
|
#include "gt/intel_engine_types.h"
|
|
|
|
|
2016-07-20 15:21:11 +07:00
|
|
|
#include "i915_gem.h"
|
2018-04-19 01:40:50 +07:00
|
|
|
#include "i915_scheduler.h"
|
2019-02-28 17:20:33 +07:00
|
|
|
#include "i915_selftest.h"
|
2016-09-09 20:11:54 +07:00
|
|
|
#include "i915_sw_fence.h"
|
2016-07-20 15:21:11 +07:00
|
|
|
|
drm/i915/scheduler: Support user-defined priorities
Use a priority stored in the context as the initial value when
submitting a request. This allows us to change the default priority on a
per-context basis, allowing different contexts to be favoured with GPU
time at the expense of lower importance work. The user can adjust the
context's priority via I915_CONTEXT_PARAM_PRIORITY, with more positive
values being higher priority (they will be serviced earlier, after their
dependencies have been resolved). Any prerequisite work for an execbuf
will have its priority raised to match the new request as required.
Normal users can specify any value in the range of -1023 to 0 [default],
i.e. they can reduce the priority of their workloads (and temporarily
boost it back to normal if so desired).
Privileged users can specify any value in the range of -1023 to 1023,
[default is 0], i.e. they can raise their priority above all overs and
so potentially starve the system.
Note that the existing schedulers are not fair, nor load balancing, the
execution is strictly by priority on a first-come, first-served basis,
and the driver may choose to boost some requests above the range
available to users.
This priority was originally based around nice(2), but evolved to allow
clients to adjust their priority within a small range, and allow for a
privileged high priority range.
For example, this can be used to implement EGL_IMG_context_priority
https://www.khronos.org/registry/egl/extensions/IMG/EGL_IMG_context_priority.txt
EGL_CONTEXT_PRIORITY_LEVEL_IMG determines the priority level of
the context to be created. This attribute is a hint, as an
implementation may not support multiple contexts at some
priority levels and system policy may limit access to high
priority contexts to appropriate system privilege level. The
default value for EGL_CONTEXT_PRIORITY_LEVEL_IMG is
EGL_CONTEXT_PRIORITY_MEDIUM_IMG."
so we can map
PRIORITY_HIGH -> 1023 [privileged, will failback to 0]
PRIORITY_MED -> 0 [default]
PRIORITY_LOW -> -1023
They also map onto the priorities used by VkQueue (and a VkQueue is
essentially a timeline, our i915_gem_context under full-ppgtt).
v2: s/CAP_SYS_ADMIN/CAP_SYS_NICE/
v3: Report min/max user priorities as defines in the uapi, and rebase
internal priorities on the exposed values.
Testcase: igt/gem_exec_schedule
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171003203453.15692-9-chris@chris-wilson.co.uk
2017-10-04 03:34:53 +07:00
|
|
|
#include <uapi/drm/i915_drm.h>
|
|
|
|
|
2016-11-11 17:43:54 +07:00
|
|
|
struct drm_file;
|
|
|
|
struct drm_i915_gem_object;
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_request;
|
2019-06-21 14:08:10 +07:00
|
|
|
struct intel_timeline;
|
|
|
|
struct intel_timeline_cacheline;
|
2016-11-11 17:43:54 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_capture_list {
|
|
|
|
struct i915_capture_list *next;
|
2017-04-15 16:39:02 +07:00
|
|
|
struct i915_vma *vma;
|
|
|
|
};
|
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
enum {
|
|
|
|
/*
|
|
|
|
* I915_FENCE_FLAG_ACTIVE - this request is currently submitted to HW.
|
|
|
|
*
|
|
|
|
* Set by __i915_request_submit() on handing over to HW, and cleared
|
|
|
|
* by __i915_request_unsubmit() if we preempt this request.
|
|
|
|
*
|
|
|
|
* Finally cleared for consistency on retiring the request, when
|
|
|
|
* we know the HW is no longer running this request.
|
|
|
|
*
|
|
|
|
* See i915_request_is_active()
|
|
|
|
*/
|
|
|
|
I915_FENCE_FLAG_ACTIVE = DMA_FENCE_FLAG_USER_BITS,
|
|
|
|
|
|
|
|
/*
|
|
|
|
* I915_FENCE_FLAG_SIGNAL - this request is currently on signal_list
|
|
|
|
*
|
|
|
|
* Internal bookkeeping used by the breadcrumb code to track when
|
|
|
|
* a request is on the various signal_list.
|
|
|
|
*/
|
|
|
|
I915_FENCE_FLAG_SIGNAL,
|
|
|
|
};
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/**
|
|
|
|
* Request queue structure.
|
|
|
|
*
|
|
|
|
* The request queue allows us to note sequence numbers that have been emitted
|
|
|
|
* and may be associated with active buffers to be retired.
|
|
|
|
*
|
|
|
|
* By keeping this list, we can avoid having to do questionable sequence
|
|
|
|
* number comparisons on buffer last_read|write_seqno. It also allows an
|
|
|
|
* emission time to be associated with the request for tracking how far ahead
|
|
|
|
* of the GPU the submission is.
|
|
|
|
*
|
2016-08-09 15:23:34 +07:00
|
|
|
* When modifying this structure be very aware that we perform a lockless
|
|
|
|
* RCU lookup of it that may race against reallocation of the struct
|
|
|
|
* from the slab freelist. We intentionally do not zero the structure on
|
|
|
|
* allocation so that the lookup can use the dangling pointers (and is
|
|
|
|
* cogniscent that those pointers may be wrong). Instead, everything that
|
|
|
|
* needs to be initialised must be done so explicitly.
|
|
|
|
*
|
2016-07-20 15:21:11 +07:00
|
|
|
* The requests are reference counted.
|
2016-07-20 15:21:08 +07:00
|
|
|
*/
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_request {
|
2016-10-25 19:00:45 +07:00
|
|
|
struct dma_fence fence;
|
2016-07-20 15:21:11 +07:00
|
|
|
spinlock_t lock;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
|
|
|
/** On Which ring this request was generated */
|
|
|
|
struct drm_i915_private *i915;
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Context and ring buffer related to this request
|
|
|
|
* Contexts are refcounted, so when this request is associated with a
|
|
|
|
* context, we must increment the context's refcount, to guarantee that
|
|
|
|
* it persists while any request is linked to it. Requests themselves
|
|
|
|
* are also refcounted, so the request will only be freed when the last
|
|
|
|
* reference to it is dismissed, and the code in
|
2018-02-21 16:56:36 +07:00
|
|
|
* i915_request_free() will then decrement the refcount on the
|
2016-07-20 15:21:08 +07:00
|
|
|
* context.
|
|
|
|
*/
|
2018-05-18 04:26:30 +07:00
|
|
|
struct i915_gem_context *gem_context;
|
2016-07-20 15:21:08 +07:00
|
|
|
struct intel_engine_cs *engine;
|
2018-05-18 04:26:32 +07:00
|
|
|
struct intel_context *hw_context;
|
2016-08-03 04:50:21 +07:00
|
|
|
struct intel_ring *ring;
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
struct intel_timeline __rcu *timeline;
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
struct list_head signal_link;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
drm/i915: Limit the backpressure for i915_request allocation
If we try and fail to allocate a i915_request, we apply some
backpressure on the clients to throttle the memory allocations coming
from i915.ko. Currently, we wait until completely idle, but this is far
too heavy and leads to some situations where the only escape is to
declare a client hung and reset the GPU. The intent is to only ratelimit
the allocation requests and to allow ourselves to recycle requests and
memory from any long queues built up by a client hog.
Although the system memory is inherently a global resources, we don't
want to overly penalize an unlucky client to pay the price of reaping a
hog. To reduce the influence of one client on another, we can instead of
waiting for the entire GPU to idle, impose a barrier on the local client.
(One end goal for request allocation is for scalability to many
concurrent allocators; simultaneous execbufs.)
To prevent ourselves from getting caught out by long running requests
(requests that may never finish without userspace intervention, whom we
are blocking) we need to impose a finite timeout, ideally shorter than
hangcheck. A long time ago Paul McKenney suggested that RCU users should
ratelimit themselves using judicious use of cond_synchronize_rcu(). This
gives us the opportunity to reduce our indefinite wait for the GPU to
idle to a wait for the RCU grace period of the previous allocation along
this timeline to expire, satisfying both the local and finite properties
we desire for our ratelimiting.
There are still a few global steps (reclaim not least amongst those!)
when we exhaust the immediate slab pool, at least now the wait is itself
decoupled from struct_mutex for our glorious highly parallel future!
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=106680
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180914080017.30308-1-chris@chris-wilson.co.uk
2018-09-14 15:00:15 +07:00
|
|
|
/*
|
|
|
|
* The rcu epoch of when this request was allocated. Used to judiciously
|
|
|
|
* apply backpressure on future allocations to ensure that under
|
|
|
|
* mempressure there is sufficient RCU ticks for us to reclaim our
|
|
|
|
* RCU protected slabs.
|
|
|
|
*/
|
|
|
|
unsigned long rcustate;
|
|
|
|
|
2019-04-03 15:21:32 +07:00
|
|
|
/*
|
|
|
|
* We pin the timeline->mutex while constructing the request to
|
|
|
|
* ensure that no caller accidentally drops it during construction.
|
|
|
|
* The timeline->mutex must be held to ensure that only this caller
|
|
|
|
* can use the ring and manipulate the associated timeline during
|
|
|
|
* construction.
|
|
|
|
*/
|
|
|
|
struct pin_cookie cookie;
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* Fences for the various phases in the request's lifetime.
|
2016-11-15 03:40:58 +07:00
|
|
|
*
|
|
|
|
* The submit fence is used to await upon all of the request's
|
|
|
|
* dependencies. When it is signaled, the request is ready to run.
|
|
|
|
* It is used by the driver to then queue the request for execution.
|
|
|
|
*/
|
2016-09-09 20:11:54 +07:00
|
|
|
struct i915_sw_fence submit;
|
2019-03-22 16:23:25 +07:00
|
|
|
union {
|
|
|
|
wait_queue_entry_t submitq;
|
|
|
|
struct i915_sw_dma_fence_cb dmaq;
|
|
|
|
};
|
2019-03-02 00:09:00 +07:00
|
|
|
struct list_head execute_cb;
|
drm/i915: Bump ready tasks ahead of busywaits
Consider two tasks that are running in parallel on a pair of engines
(vcs0, vcs1), but then must complete on a shared engine (rcs0). To
maximise throughput, we want to run the first ready task on rcs0 (i.e.
the first task that completes on either of vcs0 or vcs1). When using
semaphores, however, we will instead queue onto rcs in submission order.
To resolve this incorrect ordering, we want to re-evaluate the priority
queue when each of the request is ready. Normally this happens because
we only insert into the priority queue requests that are ready, but with
semaphores we are inserting ahead of their readiness and to compensate
we penalize those tasks with reduced priority (so that tasks that do not
need to busywait should naturally be run first). However, given a series
of tasks that each use semaphores, the queue degrades into submission
fifo rather than readiness fifo, and so to counter this we give a small
boost to semaphore users as their dependent tasks are completed (and so
we no longer require any busywait prior to running the user task as they
are then ready themselves).
v2: Fixup irqsave for schedule_lock (Tvrtko)
Testcase: igt/gem_exec_schedule/semaphore-codependency
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190409152922.23894-1-chris@chris-wilson.co.uk
2019-04-09 22:29:22 +07:00
|
|
|
struct i915_sw_fence semaphore;
|
2016-09-09 20:11:54 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* A list of everyone we wait upon, and everyone who waits upon us.
|
2016-11-15 03:41:02 +07:00
|
|
|
* Even though we will not be submitted to the hardware before the
|
|
|
|
* submit fence is signaled (it waits for all external events as well
|
|
|
|
* as our own requests), the scheduler still needs to know the
|
|
|
|
* dependency tree for the lifetime of the request (from execbuf
|
|
|
|
* to retirement), i.e. bidirectional dependency information for the
|
|
|
|
* request not tied to individual fences.
|
|
|
|
*/
|
2018-04-19 01:40:51 +07:00
|
|
|
struct i915_sched_node sched;
|
2016-11-15 03:41:02 +07:00
|
|
|
struct i915_dependency dep;
|
2019-05-22 04:11:31 +07:00
|
|
|
intel_engine_mask_t execution_mask;
|
2016-11-15 03:41:02 +07:00
|
|
|
|
2019-01-29 01:18:07 +07:00
|
|
|
/*
|
|
|
|
* A convenience pointer to the current breadcrumb value stored in
|
|
|
|
* the HW status page (or our timeline's local equivalent). The full
|
|
|
|
* path would be rq->hw_context->ring->timeline->hwsp_seqno.
|
|
|
|
*/
|
|
|
|
const u32 *hwsp_seqno;
|
|
|
|
|
2019-03-02 00:08:59 +07:00
|
|
|
/*
|
|
|
|
* If we need to access the timeline's seqno for this request in
|
|
|
|
* another request, we need to keep a read reference to this associated
|
|
|
|
* cacheline, so that we do not free and recycle it before the foreign
|
|
|
|
* observers have completed. Hence, we keep a pointer to the cacheline
|
|
|
|
* inside the timeline's HWSP vma, but it is only valid while this
|
|
|
|
* request has not completed and guarded by the timeline mutex.
|
|
|
|
*/
|
2019-06-21 14:08:10 +07:00
|
|
|
struct intel_timeline_cacheline *hwsp_cacheline;
|
2019-03-02 00:08:59 +07:00
|
|
|
|
2016-09-09 20:11:43 +07:00
|
|
|
/** Position in the ring of the start of the request */
|
2016-07-20 15:21:08 +07:00
|
|
|
u32 head;
|
|
|
|
|
2018-06-11 18:08:44 +07:00
|
|
|
/** Position in the ring of the start of the user packets */
|
|
|
|
u32 infix;
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/**
|
2016-09-09 20:11:43 +07:00
|
|
|
* Position in the ring of the start of the postfix.
|
|
|
|
* This is required to calculate the maximum available ring space
|
|
|
|
* without overwriting the postfix.
|
2016-07-20 15:21:08 +07:00
|
|
|
*/
|
|
|
|
u32 postfix;
|
|
|
|
|
2016-09-09 20:11:43 +07:00
|
|
|
/** Position in the ring of the end of the whole request */
|
2016-07-20 15:21:08 +07:00
|
|
|
u32 tail;
|
|
|
|
|
2016-09-09 20:11:43 +07:00
|
|
|
/** Position in the ring of the end of any workarounds after the tail */
|
|
|
|
u32 wa_tail;
|
|
|
|
|
|
|
|
/** Preallocate space in the ring for the emitting the request */
|
2016-07-20 15:21:08 +07:00
|
|
|
u32 reserved_space;
|
|
|
|
|
|
|
|
/** Batch buffer related to this request if any (used for
|
|
|
|
* error state dump only).
|
|
|
|
*/
|
2016-08-15 16:49:06 +07:00
|
|
|
struct i915_vma *batch;
|
2018-02-21 16:56:36 +07:00
|
|
|
/**
|
|
|
|
* Additional buffers requested by userspace to be captured upon
|
2017-04-15 16:39:02 +07:00
|
|
|
* a GPU hang. The vma/obj on this list are protected by their
|
|
|
|
* active reference - all objects on this list must also be
|
|
|
|
* on the active_list (of their final request).
|
|
|
|
*/
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_capture_list *capture_list;
|
drm/i915: Refactor activity tracking for requests
With the introduction of requests, we amplified the number of atomic
refcounted objects we use and update every execbuffer; from none to
several references, and a set of references that need to be changed. We
also introduced interesting side-effects in the order of retiring
requests and objects.
Instead of independently tracking the last request for an object, track
the active objects for each request. The object will reside in the
buffer list of its most recent active request and so we reduce the kref
interchange to a list_move. Now retirements are entirely driven by the
request, dramatically simplifying activity tracking on the object
themselves, and removing the ambiguity between retiring objects and
retiring requests.
Furthermore with the consolidation of managing the activity tracking
centrally, we can look forward to using RCU to enable lockless lookup of
the current active requests for an object. In the future, we will be
able to query the status or wait upon rendering to an object without
even touching the struct_mutex BKL.
All told, less code, simpler and faster, and more extensible.
v2: Add a typedef for the function pointer for convenience later.
v3: Make the noop retirement callback explicit. Allow passing NULL to
the init_request_active() which is expanded to a common noop function.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1470293567-10811-16-git-send-email-chris@chris-wilson.co.uk
2016-08-04 13:52:35 +07:00
|
|
|
struct list_head active_list;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
|
|
|
/** Time at which this request was emitted, in jiffies. */
|
|
|
|
unsigned long emitted_jiffies;
|
|
|
|
|
2019-07-09 23:42:27 +07:00
|
|
|
unsigned long flags;
|
|
|
|
#define I915_REQUEST_WAITBOOST BIT(0)
|
|
|
|
#define I915_REQUEST_NOPREEMPT BIT(1)
|
2017-06-28 19:35:48 +07:00
|
|
|
|
2019-06-14 23:46:06 +07:00
|
|
|
/** timeline->request entry for this request */
|
2016-08-04 13:52:33 +07:00
|
|
|
struct list_head link;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
|
|
|
struct drm_i915_file_private *file_priv;
|
|
|
|
/** file_priv list entry for this request */
|
2017-03-02 19:25:25 +07:00
|
|
|
struct list_head client_link;
|
2019-02-28 17:20:33 +07:00
|
|
|
|
|
|
|
I915_SELFTEST_DECLARE(struct {
|
|
|
|
struct list_head link;
|
|
|
|
unsigned long delay;
|
|
|
|
} mock;)
|
2016-07-20 15:21:08 +07:00
|
|
|
};
|
|
|
|
|
2017-12-13 01:06:51 +07:00
|
|
|
#define I915_FENCE_GFP (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
extern const struct dma_fence_ops i915_fence_ops;
|
2016-07-20 15:21:11 +07:00
|
|
|
|
2016-10-28 19:58:24 +07:00
|
|
|
static inline bool dma_fence_is_i915(const struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
|
|
|
return fence->ops == &i915_fence_ops;
|
|
|
|
}
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
struct i915_request * __must_check
|
|
|
|
__i915_request_create(struct intel_context *ce, gfp_t gfp);
|
|
|
|
struct i915_request * __must_check
|
|
|
|
i915_request_create(struct intel_context *ce);
|
|
|
|
|
|
|
|
struct i915_request *__i915_request_commit(struct i915_request *request);
|
2019-08-14 02:07:05 +07:00
|
|
|
void __i915_request_queue(struct i915_request *rq,
|
|
|
|
const struct i915_sched_attr *attr);
|
2019-04-25 03:07:16 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void i915_request_retire_upto(struct i915_request *rq);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
static inline struct i915_request *
|
2016-10-25 19:00:45 +07:00
|
|
|
to_request(struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
|
|
|
/* We assume that NULL fence/request are interoperable */
|
2018-02-21 16:56:36 +07:00
|
|
|
BUILD_BUG_ON(offsetof(struct i915_request, fence) != 0);
|
2016-10-28 19:58:24 +07:00
|
|
|
GEM_BUG_ON(fence && !dma_fence_is_i915(fence));
|
2018-02-21 16:56:36 +07:00
|
|
|
return container_of(fence, struct i915_request, fence);
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
static inline struct i915_request *
|
|
|
|
i915_request_get(struct i915_request *rq)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
return to_request(dma_fence_get(&rq->fence));
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
static inline struct i915_request *
|
|
|
|
i915_request_get_rcu(struct i915_request *rq)
|
drm/i915: Enable lockless lookup of request tracking via RCU
If we enable RCU for the requests (providing a grace period where we can
inspect a "dead" request before it is freed), we can allow callers to
carefully perform lockless lookup of an active request.
However, by enabling deferred freeing of requests, we can potentially
hog a lot of memory when dealing with tens of thousands of requests per
second - with a quick insertion of a synchronize_rcu() inside our
shrinker callback, that issue disappears.
v2: Currently, it is our responsibility to handle reclaim i.e. to avoid
hogging memory with the delayed slab frees. At the moment, we wait for a
grace period in the shrinker, and block for all RCU callbacks on oom.
Suggested alternatives focus on flushing our RCU callback when we have a
certain number of outstanding request frees, and blocking on that flush
after a second high watermark. (So rather than wait for the system to
run out of memory, we stop issuing requests - both are nondeterministic.)
Paul E. McKenney wrote:
Another approach is synchronize_rcu() after some largish number of
requests. The advantage of this approach is that it throttles the
production of callbacks at the source. The corresponding disadvantage
is that it slows things up.
Another approach is to use call_rcu(), but if the previous call_rcu()
is still in flight, block waiting for it. Yet another approach is
the get_state_synchronize_rcu() / cond_synchronize_rcu() pair. The
idea is to do something like this:
cond_synchronize_rcu(cookie);
cookie = get_state_synchronize_rcu();
You would of course do an initial get_state_synchronize_rcu() to
get things going. This would not block unless there was less than
one grace period's worth of time between invocations. But this
assumes a busy system, where there is almost always a grace period
in flight. But you can make that happen as follows:
cond_synchronize_rcu(cookie);
cookie = get_state_synchronize_rcu();
call_rcu(&my_rcu_head, noop_function);
Note that you need additional code to make sure that the old callback
has completed before doing a new one. Setting and clearing a flag
with appropriate memory ordering control suffices (e.g,. smp_load_acquire()
and smp_store_release()).
v3: More comments on compiler and processor order of operations within
the RCU lookup and discover we can use rcu_access_pointer() here instead.
v4: Wrap i915_gem_active_get_rcu() to take the rcu_read_lock itself.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: "Goel, Akash" <akash.goel@intel.com>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/1470324762-2545-25-git-send-email-chris@chris-wilson.co.uk
2016-08-04 22:32:41 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
return to_request(dma_fence_get_rcu(&rq->fence));
|
drm/i915: Enable lockless lookup of request tracking via RCU
If we enable RCU for the requests (providing a grace period where we can
inspect a "dead" request before it is freed), we can allow callers to
carefully perform lockless lookup of an active request.
However, by enabling deferred freeing of requests, we can potentially
hog a lot of memory when dealing with tens of thousands of requests per
second - with a quick insertion of a synchronize_rcu() inside our
shrinker callback, that issue disappears.
v2: Currently, it is our responsibility to handle reclaim i.e. to avoid
hogging memory with the delayed slab frees. At the moment, we wait for a
grace period in the shrinker, and block for all RCU callbacks on oom.
Suggested alternatives focus on flushing our RCU callback when we have a
certain number of outstanding request frees, and blocking on that flush
after a second high watermark. (So rather than wait for the system to
run out of memory, we stop issuing requests - both are nondeterministic.)
Paul E. McKenney wrote:
Another approach is synchronize_rcu() after some largish number of
requests. The advantage of this approach is that it throttles the
production of callbacks at the source. The corresponding disadvantage
is that it slows things up.
Another approach is to use call_rcu(), but if the previous call_rcu()
is still in flight, block waiting for it. Yet another approach is
the get_state_synchronize_rcu() / cond_synchronize_rcu() pair. The
idea is to do something like this:
cond_synchronize_rcu(cookie);
cookie = get_state_synchronize_rcu();
You would of course do an initial get_state_synchronize_rcu() to
get things going. This would not block unless there was less than
one grace period's worth of time between invocations. But this
assumes a busy system, where there is almost always a grace period
in flight. But you can make that happen as follows:
cond_synchronize_rcu(cookie);
cookie = get_state_synchronize_rcu();
call_rcu(&my_rcu_head, noop_function);
Note that you need additional code to make sure that the old callback
has completed before doing a new one. Setting and clearing a flag
with appropriate memory ordering control suffices (e.g,. smp_load_acquire()
and smp_store_release()).
v3: More comments on compiler and processor order of operations within
the RCU lookup and discover we can use rcu_access_pointer() here instead.
v4: Wrap i915_gem_active_get_rcu() to take the rcu_read_lock itself.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: "Goel, Akash" <akash.goel@intel.com>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/1470324762-2545-25-git-send-email-chris@chris-wilson.co.uk
2016-08-04 22:32:41 +07:00
|
|
|
}
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
static inline void
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_request_put(struct i915_request *rq)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
dma_fence_put(&rq->fence);
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
int i915_request_await_object(struct i915_request *to,
|
2016-09-09 20:11:56 +07:00
|
|
|
struct drm_i915_gem_object *obj,
|
|
|
|
bool write);
|
2018-02-21 16:56:36 +07:00
|
|
|
int i915_request_await_dma_fence(struct i915_request *rq,
|
|
|
|
struct dma_fence *fence);
|
2019-05-22 04:11:32 +07:00
|
|
|
int i915_request_await_execution(struct i915_request *rq,
|
|
|
|
struct dma_fence *fence,
|
|
|
|
void (*hook)(struct i915_request *rq,
|
|
|
|
struct dma_fence *signal));
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2018-06-12 17:51:35 +07:00
|
|
|
void i915_request_add(struct i915_request *rq);
|
2016-11-15 03:40:59 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void __i915_request_submit(struct i915_request *request);
|
|
|
|
void i915_request_submit(struct i915_request *request);
|
2017-02-23 14:44:17 +07:00
|
|
|
|
2018-07-06 17:39:43 +07:00
|
|
|
void i915_request_skip(struct i915_request *request, int error);
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void __i915_request_unsubmit(struct i915_request *request);
|
|
|
|
void i915_request_unsubmit(struct i915_request *request);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
/* Note: part of the intel_breadcrumbs family */
|
|
|
|
bool i915_request_enable_breadcrumb(struct i915_request *request);
|
|
|
|
void i915_request_cancel_breadcrumb(struct i915_request *request);
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
long i915_request_wait(struct i915_request *rq,
|
2016-10-28 19:58:27 +07:00
|
|
|
unsigned int flags,
|
|
|
|
long timeout)
|
drm/i915: Refactor activity tracking for requests
With the introduction of requests, we amplified the number of atomic
refcounted objects we use and update every execbuffer; from none to
several references, and a set of references that need to be changed. We
also introduced interesting side-effects in the order of retiring
requests and objects.
Instead of independently tracking the last request for an object, track
the active objects for each request. The object will reside in the
buffer list of its most recent active request and so we reduce the kref
interchange to a list_move. Now retirements are entirely driven by the
request, dramatically simplifying activity tracking on the object
themselves, and removing the ambiguity between retiring objects and
retiring requests.
Furthermore with the consolidation of managing the activity tracking
centrally, we can look forward to using RCU to enable lockless lookup of
the current active requests for an object. In the future, we will be
able to query the status or wait upon rendering to an object without
even touching the struct_mutex BKL.
All told, less code, simpler and faster, and more extensible.
v2: Add a typedef for the function pointer for convenience later.
v3: Make the noop retirement callback explicit. Allow passing NULL to
the init_request_active() which is expanded to a common noop function.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1470293567-10811-16-git-send-email-chris@chris-wilson.co.uk
2016-08-04 13:52:35 +07:00
|
|
|
__attribute__((nonnull(1)));
|
2016-09-09 20:11:50 +07:00
|
|
|
#define I915_WAIT_INTERRUPTIBLE BIT(0)
|
|
|
|
#define I915_WAIT_LOCKED BIT(1) /* struct_mutex held, handle GPU reset */
|
2018-10-01 21:47:55 +07:00
|
|
|
#define I915_WAIT_PRIORITY BIT(2) /* small priority bump for the request */
|
|
|
|
#define I915_WAIT_ALL BIT(3) /* used by i915_gem_object_wait() */
|
|
|
|
#define I915_WAIT_FOR_IDLE_BOOST BIT(4)
|
drm/i915: Refactor activity tracking for requests
With the introduction of requests, we amplified the number of atomic
refcounted objects we use and update every execbuffer; from none to
several references, and a set of references that need to be changed. We
also introduced interesting side-effects in the order of retiring
requests and objects.
Instead of independently tracking the last request for an object, track
the active objects for each request. The object will reside in the
buffer list of its most recent active request and so we reduce the kref
interchange to a list_move. Now retirements are entirely driven by the
request, dramatically simplifying activity tracking on the object
themselves, and removing the ambiguity between retiring objects and
retiring requests.
Furthermore with the consolidation of managing the activity tracking
centrally, we can look forward to using RCU to enable lockless lookup of
the current active requests for an object. In the future, we will be
able to query the status or wait upon rendering to an object without
even touching the struct_mutex BKL.
All told, less code, simpler and faster, and more extensible.
v2: Add a typedef for the function pointer for convenience later.
v3: Make the noop retirement callback explicit. Allow passing NULL to
the init_request_active() which is expanded to a common noop function.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1470293567-10811-16-git-send-email-chris@chris-wilson.co.uk
2016-08-04 13:52:35 +07:00
|
|
|
|
2019-01-22 05:21:02 +07:00
|
|
|
static inline bool i915_request_signaled(const struct i915_request *rq)
|
|
|
|
{
|
2019-01-29 01:18:11 +07:00
|
|
|
/* The request may live longer than its HWSP, so check flags first! */
|
2019-01-22 05:21:02 +07:00
|
|
|
return test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &rq->fence.flags);
|
|
|
|
}
|
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
static inline bool i915_request_is_active(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
return test_bit(I915_FENCE_FLAG_ACTIVE, &rq->fence.flags);
|
|
|
|
}
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/**
|
|
|
|
* Returns true if seq1 is later than seq2.
|
|
|
|
*/
|
|
|
|
static inline bool i915_seqno_passed(u32 seq1, u32 seq2)
|
|
|
|
{
|
|
|
|
return (s32)(seq1 - seq2) >= 0;
|
|
|
|
}
|
|
|
|
|
2019-01-29 01:18:07 +07:00
|
|
|
static inline u32 __hwsp_seqno(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
return READ_ONCE(*rq->hwsp_seqno);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* hwsp_seqno - the current breadcrumb value in the HW status page
|
|
|
|
* @rq: the request, to chase the relevant HW status page
|
|
|
|
*
|
|
|
|
* The emphasis in naming here is that hwsp_seqno() is not a property of the
|
|
|
|
* request, but an indication of the current HW state (associated with this
|
|
|
|
* request). Its value will change as the GPU executes more requests.
|
|
|
|
*
|
|
|
|
* Returns the current breadcrumb value in the associated HW status page (or
|
|
|
|
* the local timeline's equivalent) for this request. The request itself
|
|
|
|
* has the associated breadcrumb value of rq->fence.seqno, when the HW
|
|
|
|
* status page has that breadcrumb or later, this request is complete.
|
|
|
|
*/
|
|
|
|
static inline u32 hwsp_seqno(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
u32 seqno;
|
|
|
|
|
|
|
|
rcu_read_lock(); /* the HWSP may be freed at runtime */
|
|
|
|
seqno = __hwsp_seqno(rq);
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
|
|
|
return seqno;
|
|
|
|
}
|
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
static inline bool __i915_request_has_started(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
return i915_seqno_passed(hwsp_seqno(rq), rq->fence.seqno - 1);
|
|
|
|
}
|
|
|
|
|
2018-08-06 18:26:05 +07:00
|
|
|
/**
|
|
|
|
* i915_request_started - check if the request has begun being executed
|
|
|
|
* @rq: the request
|
|
|
|
*
|
2019-03-02 00:09:00 +07:00
|
|
|
* If the timeline is not using initial breadcrumbs, a request is
|
|
|
|
* considered started if the previous request on its timeline (i.e.
|
|
|
|
* context) has been signaled.
|
|
|
|
*
|
|
|
|
* If the timeline is using semaphores, it will also be emitting an
|
|
|
|
* "initial breadcrumb" after the semaphores are complete and just before
|
|
|
|
* it began executing the user payload. A request can therefore be active
|
|
|
|
* on the HW and not yet started as it is still busywaiting on its
|
|
|
|
* dependencies (via HW semaphores).
|
|
|
|
*
|
|
|
|
* If the request has started, its dependencies will have been signaled
|
|
|
|
* (either by fences or by semaphores) and it will have begun processing
|
|
|
|
* the user payload.
|
|
|
|
*
|
|
|
|
* However, even if a request has started, it may have been preempted and
|
|
|
|
* so no longer active, or it may have already completed.
|
|
|
|
*
|
|
|
|
* See also i915_request_is_active().
|
|
|
|
*
|
|
|
|
* Returns true if the request has begun executing the user payload, or
|
|
|
|
* has completed:
|
2018-08-06 18:26:05 +07:00
|
|
|
*/
|
|
|
|
static inline bool i915_request_started(const struct i915_request *rq)
|
|
|
|
{
|
2019-01-29 01:18:11 +07:00
|
|
|
if (i915_request_signaled(rq))
|
|
|
|
return true;
|
2018-08-06 18:26:05 +07:00
|
|
|
|
2019-01-30 01:54:50 +07:00
|
|
|
/* Remember: started but may have since been preempted! */
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
return __i915_request_has_started(rq);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* i915_request_is_running - check if the request may actually be executing
|
|
|
|
* @rq: the request
|
|
|
|
*
|
|
|
|
* Returns true if the request is currently submitted to hardware, has passed
|
|
|
|
* its start point (i.e. the context is setup and not busywaiting). Note that
|
|
|
|
* it may no longer be running by the time the function returns!
|
|
|
|
*/
|
|
|
|
static inline bool i915_request_is_running(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
if (!i915_request_is_active(rq))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
return __i915_request_has_started(rq);
|
2016-10-28 19:58:49 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
static inline bool i915_request_completed(const struct i915_request *rq)
|
2016-10-28 19:58:49 +07:00
|
|
|
{
|
2019-01-29 01:18:11 +07:00
|
|
|
if (i915_request_signaled(rq))
|
|
|
|
return true;
|
2017-02-23 14:44:14 +07:00
|
|
|
|
2019-01-29 01:18:11 +07:00
|
|
|
return i915_seqno_passed(hwsp_seqno(rq), rq->fence.seqno);
|
|
|
|
}
|
2016-10-28 19:58:49 +07:00
|
|
|
|
2019-01-29 01:18:11 +07:00
|
|
|
static inline void i915_request_mark_complete(struct i915_request *rq)
|
|
|
|
{
|
|
|
|
rq->hwsp_seqno = (u32 *)&rq->fence.seqno; /* decouple from HWSP */
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2019-07-09 23:42:27 +07:00
|
|
|
static inline bool i915_request_has_waitboost(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
return rq->flags & I915_REQUEST_WAITBOOST;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline bool i915_request_has_nopreempt(const struct i915_request *rq)
|
|
|
|
{
|
|
|
|
/* Preemption should only be disabled very rarely */
|
|
|
|
return unlikely(rq->flags & I915_REQUEST_NOPREEMPT);
|
|
|
|
}
|
|
|
|
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
static inline struct intel_timeline *
|
|
|
|
i915_request_timeline(struct i915_request *rq)
|
|
|
|
{
|
|
|
|
/* Valid only while the request is being constructed (or retired). */
|
|
|
|
return rcu_dereference_protected(rq->timeline,
|
|
|
|
lockdep_is_held(&rcu_access_pointer(rq->timeline)->mutex));
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct intel_timeline *
|
|
|
|
i915_request_active_timeline(struct i915_request *rq)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* When in use during submission, we are protected by a guarantee that
|
|
|
|
* the context/timeline is pinned and must remain pinned until after
|
|
|
|
* this submission.
|
|
|
|
*/
|
|
|
|
return rcu_dereference_protected(rq->timeline,
|
|
|
|
lockdep_is_held(&rq->engine->active.lock));
|
|
|
|
}
|
|
|
|
|
drm/i915: Invert the GEM wakeref hierarchy
In the current scheme, on submitting a request we take a single global
GEM wakeref, which trickles down to wake up all GT power domains. This
is undesirable as we would like to be able to localise our power
management to the available power domains and to remove the global GEM
operations from the heart of the driver. (The intent there is to push
global GEM decisions to the boundary as used by the GEM user interface.)
Now during request construction, each request is responsible via its
logical context to acquire a wakeref on each power domain it intends to
utilize. Currently, each request takes a wakeref on the engine(s) and
the engines themselves take a chipset wakeref. This gives us a
transition on each engine which we can extend if we want to insert more
powermangement control (such as soft rc6). The global GEM operations
that currently require a struct_mutex are reduced to listening to pm
events from the chipset GT wakeref. As we reduce the struct_mutex
requirement, these listeners should evaporate.
Perhaps the biggest immediate change is that this removes the
struct_mutex requirement around GT power management, allowing us greater
flexibility in request construction. Another important knock-on effect,
is that by tracking engine usage, we can insert a switch back to the
kernel context on that engine immediately, avoiding any extra delay or
inserting global synchronisation barriers. This makes tracking when an
engine and its associated contexts are idle much easier -- important for
when we forgo our assumed execution ordering and need idle barriers to
unpin used contexts. In the process, it means we remove a large chunk of
code whose only purpose was to switch back to the kernel context.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Imre Deak <imre.deak@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190424200717.1686-5-chris@chris-wilson.co.uk
2019-04-25 03:07:17 +07:00
|
|
|
bool i915_retire_requests(struct drm_i915_private *i915);
|
2018-02-21 16:56:36 +07:00
|
|
|
|
|
|
|
#endif /* I915_REQUEST_H */
|