2016-07-20 15:21:08 +07:00
|
|
|
/*
|
|
|
|
* Copyright © 2008-2015 Intel Corporation
|
|
|
|
*
|
|
|
|
* Permission is hereby granted, free of charge, to any person obtaining a
|
|
|
|
* copy of this software and associated documentation files (the "Software"),
|
|
|
|
* to deal in the Software without restriction, including without limitation
|
|
|
|
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
|
|
|
|
* and/or sell copies of the Software, and to permit persons to whom the
|
|
|
|
* Software is furnished to do so, subject to the following conditions:
|
|
|
|
*
|
|
|
|
* The above copyright notice and this permission notice (including the next
|
|
|
|
* paragraph) shall be included in all copies or substantial portions of the
|
|
|
|
* Software.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
|
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
|
|
|
|
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
|
|
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
|
|
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
|
|
|
|
* IN THE SOFTWARE.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
2016-10-28 19:58:24 +07:00
|
|
|
#include <linux/dma-fence-array.h>
|
2019-03-02 00:09:00 +07:00
|
|
|
#include <linux/irq_work.h>
|
|
|
|
#include <linux/prefetch.h>
|
2017-02-01 22:36:40 +07:00
|
|
|
#include <linux/sched.h>
|
|
|
|
#include <linux/sched/clock.h>
|
2017-02-04 05:47:37 +07:00
|
|
|
#include <linux/sched/signal.h>
|
drm/i915: Refactor activity tracking for requests
With the introduction of requests, we amplified the number of atomic
refcounted objects we use and update every execbuffer; from none to
several references, and a set of references that need to be changed. We
also introduced interesting side-effects in the order of retiring
requests and objects.
Instead of independently tracking the last request for an object, track
the active objects for each request. The object will reside in the
buffer list of its most recent active request and so we reduce the kref
interchange to a list_move. Now retirements are entirely driven by the
request, dramatically simplifying activity tracking on the object
themselves, and removing the ambiguity between retiring objects and
retiring requests.
Furthermore with the consolidation of managing the activity tracking
centrally, we can look forward to using RCU to enable lockless lookup of
the current active requests for an object. In the future, we will be
able to query the status or wait upon rendering to an object without
even touching the struct_mutex BKL.
All told, less code, simpler and faster, and more extensible.
v2: Add a typedef for the function pointer for convenience later.
v3: Make the noop retirement callback explicit. Allow passing NULL to
the init_request_active() which is expanded to a common noop function.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1470293567-10811-16-git-send-email-chris@chris-wilson.co.uk
2016-08-04 13:52:35 +07:00
|
|
|
|
2019-05-28 16:29:49 +07:00
|
|
|
#include "gem/i915_gem_context.h"
|
|
|
|
#include "gt/intel_context.h"
|
2019-10-24 17:03:44 +07:00
|
|
|
#include "gt/intel_ring.h"
|
2019-10-25 04:16:41 +07:00
|
|
|
#include "gt/intel_rps.h"
|
2019-05-28 16:29:49 +07:00
|
|
|
|
drm/i915: Pull i915_gem_active into the i915_active family
Looking forward, we need to break the struct_mutex dependency on
i915_gem_active. In the meantime, external use of i915_gem_active is
quite beguiling, little do new users suspect that it implies a barrier
as each request it tracks must be ordered wrt the previous one. As one
of many, it can be used to track activity across multiple timelines, a
shared fence, which fits our unordered request submission much better. We
need to steer external users away from the singular, exclusive fence
imposed by i915_gem_active to i915_active instead. As part of that
process, we move i915_gem_active out of i915_request.c into
i915_active.c to start separating the two concepts, and rename it to
i915_active_request (both to tie it to the concept of tracking just one
request, and to give it a longer, less appealing name).
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190205130005.2807-5-chris@chris-wilson.co.uk
2019-02-05 20:00:05 +07:00
|
|
|
#include "i915_active.h"
|
2019-04-05 18:00:15 +07:00
|
|
|
#include "i915_drv.h"
|
2019-03-06 04:38:30 +07:00
|
|
|
#include "i915_globals.h"
|
2019-08-06 17:07:28 +07:00
|
|
|
#include "i915_trace.h"
|
2019-04-05 18:00:15 +07:00
|
|
|
#include "intel_pm.h"
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
struct execute_cb {
|
|
|
|
struct list_head link;
|
|
|
|
struct irq_work work;
|
|
|
|
struct i915_sw_fence *fence;
|
2019-05-22 04:11:32 +07:00
|
|
|
void (*hook)(struct i915_request *rq, struct dma_fence *signal);
|
|
|
|
struct i915_request *signal;
|
2019-03-02 00:09:00 +07:00
|
|
|
};
|
|
|
|
|
2019-02-28 17:20:33 +07:00
|
|
|
static struct i915_global_request {
|
2019-03-06 04:38:30 +07:00
|
|
|
struct i915_global base;
|
2019-02-28 17:20:33 +07:00
|
|
|
struct kmem_cache *slab_requests;
|
|
|
|
struct kmem_cache *slab_dependencies;
|
2019-03-02 00:09:00 +07:00
|
|
|
struct kmem_cache *slab_execute_cbs;
|
2019-02-28 17:20:33 +07:00
|
|
|
} global;
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
static const char *i915_fence_get_driver_name(struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
2019-12-11 22:02:04 +07:00
|
|
|
return dev_name(to_request(fence)->i915->drm.dev);
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
static const char *i915_fence_get_timeline_name(struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
2019-12-20 17:12:29 +07:00
|
|
|
const struct i915_gem_context *ctx;
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* The timeline struct (as part of the ppgtt underneath a context)
|
2017-03-30 18:16:14 +07:00
|
|
|
* may be freed when the request is no longer in use by the GPU.
|
|
|
|
* We could extend the life of a context to beyond that of all
|
|
|
|
* fences, possibly keeping the hw resource around indefinitely,
|
|
|
|
* or we just give them a false name. Since
|
|
|
|
* dma_fence_ops.get_timeline_name is a debug feature, the occasional
|
|
|
|
* lie seems justifiable.
|
|
|
|
*/
|
|
|
|
if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
|
|
|
|
return "signaled";
|
|
|
|
|
2019-12-20 17:12:29 +07:00
|
|
|
ctx = to_request(fence)->context->gem_context;
|
|
|
|
if (!ctx)
|
|
|
|
return "[" DRIVER_NAME "]";
|
|
|
|
|
|
|
|
return ctx->name;
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
static bool i915_fence_signaled(struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
return i915_request_completed(to_request(fence));
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
static bool i915_fence_enable_signaling(struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
return i915_request_enable_breadcrumb(to_request(fence));
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
static signed long i915_fence_wait(struct dma_fence *fence,
|
2016-07-20 15:21:11 +07:00
|
|
|
bool interruptible,
|
2016-10-28 19:58:27 +07:00
|
|
|
signed long timeout)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
2019-02-13 16:25:04 +07:00
|
|
|
return i915_request_wait(to_request(fence),
|
|
|
|
interruptible | I915_WAIT_PRIORITY,
|
|
|
|
timeout);
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
static void i915_fence_release(struct dma_fence *fence)
|
2016-07-20 15:21:11 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_request *rq = to_request(fence);
|
2016-07-20 15:21:11 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* The request is put onto a RCU freelist (i.e. the address
|
2016-11-25 20:17:18 +07:00
|
|
|
* is immediately reused), mark the fences as being freed now.
|
|
|
|
* Otherwise the debugobjects for the fences are only marked as
|
|
|
|
* freed when the slab cache itself is freed, and so we would get
|
|
|
|
* caught trying to reuse dead objects.
|
|
|
|
*/
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_sw_fence_fini(&rq->submit);
|
2019-04-11 19:24:45 +07:00
|
|
|
i915_sw_fence_fini(&rq->semaphore);
|
2016-11-25 20:17:18 +07:00
|
|
|
|
2019-02-28 17:20:33 +07:00
|
|
|
kmem_cache_free(global.slab_requests, rq);
|
2016-07-20 15:21:11 +07:00
|
|
|
}
|
|
|
|
|
2016-10-25 19:00:45 +07:00
|
|
|
const struct dma_fence_ops i915_fence_ops = {
|
2016-07-20 15:21:11 +07:00
|
|
|
.get_driver_name = i915_fence_get_driver_name,
|
|
|
|
.get_timeline_name = i915_fence_get_timeline_name,
|
|
|
|
.enable_signaling = i915_fence_enable_signaling,
|
|
|
|
.signaled = i915_fence_signaled,
|
|
|
|
.wait = i915_fence_wait,
|
|
|
|
.release = i915_fence_release,
|
|
|
|
};
|
|
|
|
|
2019-06-18 14:41:31 +07:00
|
|
|
static void irq_execute_cb(struct irq_work *wrk)
|
|
|
|
{
|
|
|
|
struct execute_cb *cb = container_of(wrk, typeof(*cb), work);
|
|
|
|
|
|
|
|
i915_sw_fence_complete(cb->fence);
|
|
|
|
kmem_cache_free(global.slab_execute_cbs, cb);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void irq_execute_cb_hook(struct irq_work *wrk)
|
|
|
|
{
|
|
|
|
struct execute_cb *cb = container_of(wrk, typeof(*cb), work);
|
|
|
|
|
|
|
|
cb->hook(container_of(cb->fence, struct i915_request, submit),
|
|
|
|
&cb->signal->fence);
|
|
|
|
i915_request_put(cb->signal);
|
|
|
|
|
|
|
|
irq_execute_cb(wrk);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __notify_execute_cb(struct i915_request *rq)
|
|
|
|
{
|
|
|
|
struct execute_cb *cb;
|
|
|
|
|
|
|
|
lockdep_assert_held(&rq->lock);
|
|
|
|
|
|
|
|
if (list_empty(&rq->execute_cb))
|
|
|
|
return;
|
|
|
|
|
|
|
|
list_for_each_entry(cb, &rq->execute_cb, link)
|
|
|
|
irq_work_queue(&cb->work);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* XXX Rollback on __i915_request_unsubmit()
|
|
|
|
*
|
|
|
|
* In the future, perhaps when we have an active time-slicing scheduler,
|
|
|
|
* it will be interesting to unsubmit parallel execution and remove
|
|
|
|
* busywaits from the GPU until their master is restarted. This is
|
|
|
|
* quite hairy, we have to carefully rollback the fence and do a
|
|
|
|
* preempt-to-idle cycle on the target engine, all the while the
|
|
|
|
* master execute_cb may refire.
|
|
|
|
*/
|
|
|
|
INIT_LIST_HEAD(&rq->execute_cb);
|
|
|
|
}
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
static inline void
|
2019-08-20 15:09:07 +07:00
|
|
|
remove_from_client(struct i915_request *request)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2017-03-02 19:25:25 +07:00
|
|
|
struct drm_i915_file_private *file_priv;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-08-24 01:14:55 +07:00
|
|
|
if (!READ_ONCE(request->file_priv))
|
2016-07-20 15:21:08 +07:00
|
|
|
return;
|
|
|
|
|
2019-08-24 01:14:55 +07:00
|
|
|
rcu_read_lock();
|
|
|
|
file_priv = xchg(&request->file_priv, NULL);
|
|
|
|
if (file_priv) {
|
|
|
|
spin_lock(&file_priv->mm.lock);
|
2017-03-02 19:25:25 +07:00
|
|
|
list_del(&request->client_link);
|
2019-08-24 01:14:55 +07:00
|
|
|
spin_unlock(&file_priv->mm.lock);
|
2017-03-02 19:25:25 +07:00
|
|
|
}
|
2019-08-24 01:14:55 +07:00
|
|
|
rcu_read_unlock();
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
static void free_capture_list(struct i915_request *request)
|
2017-04-15 16:39:02 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_capture_list *capture;
|
2017-04-15 16:39:02 +07:00
|
|
|
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
capture = fetch_and_zero(&request->capture_list);
|
2017-04-15 16:39:02 +07:00
|
|
|
while (capture) {
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_capture_list *next = capture->next;
|
2017-04-15 16:39:02 +07:00
|
|
|
|
|
|
|
kfree(capture);
|
|
|
|
capture = next;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-09-18 21:54:50 +07:00
|
|
|
static void remove_from_engine(struct i915_request *rq)
|
|
|
|
{
|
|
|
|
struct intel_engine_cs *engine, *locked;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Virtual engines complicate acquiring the engine timeline lock,
|
|
|
|
* as their rq->engine pointer is not stable until under that
|
|
|
|
* engine lock. The simple ploy we use is to take the lock then
|
|
|
|
* check that the rq still belongs to the newly locked engine.
|
|
|
|
*/
|
|
|
|
locked = READ_ONCE(rq->engine);
|
2019-10-17 23:13:52 +07:00
|
|
|
spin_lock_irq(&locked->active.lock);
|
2019-09-18 21:54:50 +07:00
|
|
|
while (unlikely(locked != (engine = READ_ONCE(rq->engine)))) {
|
|
|
|
spin_unlock(&locked->active.lock);
|
|
|
|
spin_lock(&engine->active.lock);
|
|
|
|
locked = engine;
|
|
|
|
}
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
list_del_init(&rq->sched.link);
|
2019-10-17 23:13:52 +07:00
|
|
|
spin_unlock_irq(&locked->active.lock);
|
2019-09-18 21:54:50 +07:00
|
|
|
}
|
|
|
|
|
2019-10-04 20:40:06 +07:00
|
|
|
bool i915_request_retire(struct i915_request *rq)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2019-06-14 23:46:05 +07:00
|
|
|
if (!i915_request_completed(rq))
|
|
|
|
return false;
|
2018-03-15 20:14:50 +07:00
|
|
|
|
2019-12-13 22:51:52 +07:00
|
|
|
RQ_TRACE(rq, "\n");
|
2016-10-28 19:58:32 +07:00
|
|
|
|
2019-06-14 23:46:05 +07:00
|
|
|
GEM_BUG_ON(!i915_sw_fence_signaled(&rq->submit));
|
|
|
|
trace_i915_request_retire(rq);
|
2016-10-28 19:58:58 +07:00
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
/*
|
|
|
|
* We know the GPU must have read the request to have
|
|
|
|
* sent us the seqno + interrupt, so use the position
|
|
|
|
* of tail of the request to update the last known position
|
|
|
|
* of the GPU head.
|
|
|
|
*
|
|
|
|
* Note this requires that we are always called in request
|
|
|
|
* completion order.
|
|
|
|
*/
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
GEM_BUG_ON(!list_is_first(&rq->link,
|
|
|
|
&i915_request_timeline(rq)->requests));
|
2019-08-16 03:57:09 +07:00
|
|
|
rq->ring->head = rq->postfix;
|
2017-04-15 16:39:02 +07:00
|
|
|
|
drm/i915/execlists: Preempt-to-busy
When using a global seqno, we required a precise stop-the-workd event to
handle preemption and unwind the global seqno counter. To accomplish
this, we would preempt to a special out-of-band context and wait for the
machine to report that it was idle. Given an idle machine, we could very
precisely see which requests had completed and which we needed to feed
back into the run queue.
However, now that we have scrapped the global seqno, we no longer need
to precisely unwind the global counter and only track requests by their
per-context seqno. This allows us to loosely unwind inflight requests
while scheduling a preemption, with the enormous caveat that the
requests we put back on the run queue are still _inflight_ (until the
preemption request is complete). This makes request tracking much more
messy, as at any point then we can see a completed request that we
believe is not currently scheduled for execution. We also have to be
careful not to rewind RING_TAIL past RING_HEAD on preempting to the
running context, and for this we use a semaphore to prevent completion
of the request before continuing.
To accomplish this feat, we change how we track requests scheduled to
the HW. Instead of appending our requests onto a single list as we
submit, we track each submission to ELSP as its own block. Then upon
receiving the CS preemption event, we promote the pending block to the
inflight block (discarding what was previously being tracked). As normal
CS completion events arrive, we then remove stale entries from the
inflight tracker.
v2: Be a tinge paranoid and ensure we flush the write into the HWS page
for the GPU semaphore to pick in a timely fashion.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190620142052.19311-1-chris@chris-wilson.co.uk
2019-06-20 21:20:51 +07:00
|
|
|
/*
|
|
|
|
* We only loosely track inflight requests across preemption,
|
|
|
|
* and so we may find ourselves attempting to retire a _completed_
|
|
|
|
* request that we have removed from the HW and put back on a run
|
|
|
|
* queue.
|
|
|
|
*/
|
2019-09-18 21:54:50 +07:00
|
|
|
remove_from_engine(rq);
|
2016-11-15 03:41:02 +07:00
|
|
|
|
2019-10-17 23:13:52 +07:00
|
|
|
spin_lock_irq(&rq->lock);
|
2019-06-14 23:46:05 +07:00
|
|
|
i915_request_mark_complete(rq);
|
|
|
|
if (!i915_request_signaled(rq))
|
|
|
|
dma_fence_signal_locked(&rq->fence);
|
|
|
|
if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &rq->fence.flags))
|
|
|
|
i915_request_cancel_breadcrumb(rq);
|
2019-07-09 23:42:27 +07:00
|
|
|
if (i915_request_has_waitboost(rq)) {
|
2019-10-25 04:16:41 +07:00
|
|
|
GEM_BUG_ON(!atomic_read(&rq->engine->gt->rps.num_waiters));
|
|
|
|
atomic_dec(&rq->engine->gt->rps.num_waiters);
|
2019-06-14 23:46:05 +07:00
|
|
|
}
|
2019-06-18 14:41:31 +07:00
|
|
|
if (!test_bit(I915_FENCE_FLAG_ACTIVE, &rq->fence.flags)) {
|
|
|
|
set_bit(I915_FENCE_FLAG_ACTIVE, &rq->fence.flags);
|
|
|
|
__notify_execute_cb(rq);
|
|
|
|
}
|
|
|
|
GEM_BUG_ON(!list_empty(&rq->execute_cb));
|
2019-10-17 23:13:52 +07:00
|
|
|
spin_unlock_irq(&rq->lock);
|
2018-04-30 20:15:00 +07:00
|
|
|
|
2019-08-20 15:09:07 +07:00
|
|
|
remove_from_client(rq);
|
2019-06-14 23:46:06 +07:00
|
|
|
list_del(&rq->link);
|
2019-06-14 23:46:05 +07:00
|
|
|
|
2019-12-20 17:12:29 +07:00
|
|
|
intel_context_exit(rq->context);
|
|
|
|
intel_context_unpin(rq->context);
|
2019-08-10 01:25:18 +07:00
|
|
|
|
2019-06-14 23:46:05 +07:00
|
|
|
free_capture_list(rq);
|
|
|
|
i915_sched_node_fini(&rq->sched);
|
|
|
|
i915_request_put(rq);
|
|
|
|
|
|
|
|
return true;
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void i915_request_retire_upto(struct i915_request *rq)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
struct intel_timeline * const tl = i915_request_timeline(rq);
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_request *tmp;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-12-13 22:51:52 +07:00
|
|
|
RQ_TRACE(rq, "\n");
|
2018-04-30 20:15:02 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
GEM_BUG_ON(!i915_request_completed(rq));
|
2016-11-25 20:17:15 +07:00
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
do {
|
2019-08-16 03:57:09 +07:00
|
|
|
tmp = list_first_entry(&tl->requests, typeof(*tmp), link);
|
2019-06-14 23:46:05 +07:00
|
|
|
} while (i915_request_retire(tmp) && tmp != rq);
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
static int
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
__await_execution(struct i915_request *rq,
|
|
|
|
struct i915_request *signal,
|
|
|
|
void (*hook)(struct i915_request *rq,
|
|
|
|
struct dma_fence *signal),
|
|
|
|
gfp_t gfp)
|
2019-03-02 00:09:00 +07:00
|
|
|
{
|
|
|
|
struct execute_cb *cb;
|
|
|
|
|
2019-05-22 04:11:32 +07:00
|
|
|
if (i915_request_is_active(signal)) {
|
|
|
|
if (hook)
|
|
|
|
hook(rq, &signal->fence);
|
2019-03-02 00:09:00 +07:00
|
|
|
return 0;
|
2019-05-22 04:11:32 +07:00
|
|
|
}
|
2019-03-02 00:09:00 +07:00
|
|
|
|
|
|
|
cb = kmem_cache_alloc(global.slab_execute_cbs, gfp);
|
|
|
|
if (!cb)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
cb->fence = &rq->submit;
|
|
|
|
i915_sw_fence_await(cb->fence);
|
|
|
|
init_irq_work(&cb->work, irq_execute_cb);
|
|
|
|
|
2019-05-22 04:11:32 +07:00
|
|
|
if (hook) {
|
|
|
|
cb->hook = hook;
|
|
|
|
cb->signal = i915_request_get(signal);
|
|
|
|
cb->work.func = irq_execute_cb_hook;
|
|
|
|
}
|
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
spin_lock_irq(&signal->lock);
|
|
|
|
if (i915_request_is_active(signal)) {
|
2019-05-22 04:11:32 +07:00
|
|
|
if (hook) {
|
|
|
|
hook(rq, &signal->fence);
|
|
|
|
i915_request_put(signal);
|
|
|
|
}
|
2019-03-02 00:09:00 +07:00
|
|
|
i915_sw_fence_complete(cb->fence);
|
|
|
|
kmem_cache_free(global.slab_execute_cbs, cb);
|
|
|
|
} else {
|
|
|
|
list_add_tail(&cb->link, &signal->execute_cb);
|
|
|
|
}
|
|
|
|
spin_unlock_irq(&signal->lock);
|
|
|
|
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
/* Copy across semaphore status as we need the same behaviour */
|
|
|
|
rq->sched.flags |= signal->sched.flags;
|
2019-03-02 00:09:00 +07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2019-09-23 18:00:55 +07:00
|
|
|
bool __i915_request_submit(struct i915_request *request)
|
2016-09-09 20:11:54 +07:00
|
|
|
{
|
2016-10-28 19:58:46 +07:00
|
|
|
struct intel_engine_cs *engine = request->engine;
|
2019-09-23 18:00:55 +07:00
|
|
|
bool result = false;
|
2016-09-09 20:11:54 +07:00
|
|
|
|
2019-12-13 22:51:52 +07:00
|
|
|
RQ_TRACE(request, "\n");
|
2018-03-15 20:14:50 +07:00
|
|
|
|
2017-03-02 18:51:30 +07:00
|
|
|
GEM_BUG_ON(!irqs_disabled());
|
2019-06-14 23:46:06 +07:00
|
|
|
lockdep_assert_held(&engine->active.lock);
|
2017-03-02 18:51:30 +07:00
|
|
|
|
2019-09-23 18:00:55 +07:00
|
|
|
/*
|
|
|
|
* With the advent of preempt-to-busy, we frequently encounter
|
|
|
|
* requests that we have unsubmitted from HW, but left running
|
|
|
|
* until the next ack and so have completed in the meantime. On
|
|
|
|
* resubmission of that completed request, we can skip
|
|
|
|
* updating the payload, and execlists can even skip submitting
|
|
|
|
* the request.
|
|
|
|
*
|
|
|
|
* We must remove the request from the caller's priority queue,
|
|
|
|
* and the caller must only call us when the request is in their
|
|
|
|
* priority queue, under the active.lock. This ensures that the
|
|
|
|
* request has *not* yet been retired and we can safely move
|
|
|
|
* the request into the engine->active.list where it will be
|
|
|
|
* dropped upon retiring. (Otherwise if resubmit a *retired*
|
|
|
|
* request, this would be a horrible use-after-free.)
|
|
|
|
*/
|
|
|
|
if (i915_request_completed(request))
|
|
|
|
goto xfer;
|
|
|
|
|
2019-12-20 17:12:29 +07:00
|
|
|
if (intel_context_is_banned(request->context))
|
2019-02-14 01:27:37 +07:00
|
|
|
i915_request_skip(request, -EIO);
|
|
|
|
|
drm/i915: Disable semaphore busywaits on saturated systems
Asking the GPU to busywait on a memory address, perhaps not unexpectedly
in hindsight for a shared system, leads to bus contention that affects
CPU programs trying to concurrently access memory. This can manifest as
a drop in transcode throughput on highly over-saturated workloads.
The only clue offered by perf, is that the bus-cycles (perf stat -e
bus-cycles) jumped by 50% when enabling semaphores. This corresponds
with extra CPU active cycles being attributed to intel_idle's mwait.
This patch introduces a heuristic to try and detect when more than one
client is submitting to the GPU pushing it into an oversaturated state.
As we already keep track of when the semaphores are signaled, we can
inspect their state on submitting the busywait batch and if we planned
to use a semaphore but were too late, conclude that the GPU is
overloaded and not try to use semaphores in future requests. In
practice, this means we optimistically try to use semaphores for the
first frame of a transcode job split over multiple engines, and fail if
there are multiple clients active and continue not to use semaphores for
the subsequent frames in the sequence. Periodically, we try to
optimistically switch semaphores back on whenever the client waits to
catch up with the transcode results.
With 1 client, on Broxton J3455, with the relative fps normalized by %cpu:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| * |
| *+ |
| **+ |
| **+ x |
| x * +**+ x |
| x x * * +***x xx |
| x x * * *+***x *x |
| x x* + * * *****x *x x |
| + x xx+x* + *** * ********* x * |
| + x xx+x* * *** +** ********* xx * |
| * + ++++* + x*x****+*+* ***+*************+x* * |
|*+ +** *+ + +* + *++****** *xxx**********x***+*****************+*++ *|
| |__________A_____M_____| |
| |_______________A____M_________| |
| |____________A___M________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.60475 3.50941 3.31123 3.2143953 0.21117399
+ 120 2.3826 3.57077 3.25101 3.1414161 0.28146407
Difference at 95.0% confidence
-0.0729792 +/- 0.0629585
-2.27039% +/- 1.95864%
(Student's t, pooled s = 0.248814)
* 120 2.35536 3.66713 3.2849 3.2059917 0.24618565
No difference proven at 95.0% confidence
With 10 clients over-saturating the pipeline:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| ++ ** |
| ++ ** |
| ++ ** |
| ++ ** |
| ++ xx *** |
| ++ xx *** |
| ++ xxx*** |
| ++ xxx*** |
| +++ xxx*** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| ++++ xx**** |
| +++++ xx**** |
| +++++ x x****** |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xx******** |
| ++++++ xxxx******** |
| ++++++ xxxx******** |
| ++++++++ xxxxx********* |
|+ + + + ++++++++ xxx*xx**********x* *|
| |__A__| |
| |__AM__| |
| |__A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.47855 2.8972 2.72376 2.7193402 0.074604933
+ 120 1.17367 1.77459 1.71977 1.6966782 0.085850697
Difference at 95.0% confidence
-1.02266 +/- 0.0203502
-37.607% +/- 0.748352%
(Student's t, pooled s = 0.0804246)
* 120 2.57868 3.00821 2.80142 2.7923878 0.058646477
Difference at 95.0% confidence
0.0730476 +/- 0.0169791
2.68622% +/- 0.624383%
(Student's t, pooled s = 0.0671018)
Indicating that we've recovered the regression from enabling semaphores
on this saturated setup, with a hint towards an overall improvement.
Very similar, but of smaller magnitude, results are observed on both
Skylake(gt2) and Kabylake(gt4). This may be due to the reduced impact of
bus-cycles, where we see a 50% hit on Broxton, it is only 10% on the big
core, in this particular test.
One observation to make here is that for a greedy client trying to
maximise its own throughput, using semaphores is the right choice. It is
only the holistic system-wide view that semaphores of one client
impacts another and reduces the overall throughput where we would choose
to disable semaphores.
The most noticeable negactive impact this has is on the no-op
microbenchmarks, which are also very notable for having no cpu bus load.
In particular, this increases the runtime and energy consumption of
gem_exec_whisper.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190504070707.30902-1-chris@chris-wilson.co.uk
2019-05-04 14:07:07 +07:00
|
|
|
/*
|
|
|
|
* Are we using semaphores when the gpu is already saturated?
|
|
|
|
*
|
|
|
|
* Using semaphores incurs a cost in having the GPU poll a
|
|
|
|
* memory location, busywaiting for it to change. The continual
|
|
|
|
* memory reads can have a noticeable impact on the rest of the
|
|
|
|
* system with the extra bus traffic, stalling the cpu as it too
|
|
|
|
* tries to access memory across the bus (perf stat -e bus-cycles).
|
|
|
|
*
|
|
|
|
* If we installed a semaphore on this request and we only submit
|
|
|
|
* the request after the signaler completed, that indicates the
|
|
|
|
* system is overloaded and using semaphores at this time only
|
|
|
|
* increases the amount of work we are doing. If so, we disable
|
|
|
|
* further use of semaphores until we are idle again, whence we
|
|
|
|
* optimistically try again.
|
|
|
|
*/
|
|
|
|
if (request->sched.semaphores &&
|
|
|
|
i915_sw_fence_signaled(&request->semaphore))
|
2019-06-18 14:41:35 +07:00
|
|
|
engine->saturated |= request->sched.semaphores;
|
drm/i915: Disable semaphore busywaits on saturated systems
Asking the GPU to busywait on a memory address, perhaps not unexpectedly
in hindsight for a shared system, leads to bus contention that affects
CPU programs trying to concurrently access memory. This can manifest as
a drop in transcode throughput on highly over-saturated workloads.
The only clue offered by perf, is that the bus-cycles (perf stat -e
bus-cycles) jumped by 50% when enabling semaphores. This corresponds
with extra CPU active cycles being attributed to intel_idle's mwait.
This patch introduces a heuristic to try and detect when more than one
client is submitting to the GPU pushing it into an oversaturated state.
As we already keep track of when the semaphores are signaled, we can
inspect their state on submitting the busywait batch and if we planned
to use a semaphore but were too late, conclude that the GPU is
overloaded and not try to use semaphores in future requests. In
practice, this means we optimistically try to use semaphores for the
first frame of a transcode job split over multiple engines, and fail if
there are multiple clients active and continue not to use semaphores for
the subsequent frames in the sequence. Periodically, we try to
optimistically switch semaphores back on whenever the client waits to
catch up with the transcode results.
With 1 client, on Broxton J3455, with the relative fps normalized by %cpu:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| * |
| *+ |
| **+ |
| **+ x |
| x * +**+ x |
| x x * * +***x xx |
| x x * * *+***x *x |
| x x* + * * *****x *x x |
| + x xx+x* + *** * ********* x * |
| + x xx+x* * *** +** ********* xx * |
| * + ++++* + x*x****+*+* ***+*************+x* * |
|*+ +** *+ + +* + *++****** *xxx**********x***+*****************+*++ *|
| |__________A_____M_____| |
| |_______________A____M_________| |
| |____________A___M________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.60475 3.50941 3.31123 3.2143953 0.21117399
+ 120 2.3826 3.57077 3.25101 3.1414161 0.28146407
Difference at 95.0% confidence
-0.0729792 +/- 0.0629585
-2.27039% +/- 1.95864%
(Student's t, pooled s = 0.248814)
* 120 2.35536 3.66713 3.2849 3.2059917 0.24618565
No difference proven at 95.0% confidence
With 10 clients over-saturating the pipeline:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| ++ ** |
| ++ ** |
| ++ ** |
| ++ ** |
| ++ xx *** |
| ++ xx *** |
| ++ xxx*** |
| ++ xxx*** |
| +++ xxx*** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| ++++ xx**** |
| +++++ xx**** |
| +++++ x x****** |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xx******** |
| ++++++ xxxx******** |
| ++++++ xxxx******** |
| ++++++++ xxxxx********* |
|+ + + + ++++++++ xxx*xx**********x* *|
| |__A__| |
| |__AM__| |
| |__A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.47855 2.8972 2.72376 2.7193402 0.074604933
+ 120 1.17367 1.77459 1.71977 1.6966782 0.085850697
Difference at 95.0% confidence
-1.02266 +/- 0.0203502
-37.607% +/- 0.748352%
(Student's t, pooled s = 0.0804246)
* 120 2.57868 3.00821 2.80142 2.7923878 0.058646477
Difference at 95.0% confidence
0.0730476 +/- 0.0169791
2.68622% +/- 0.624383%
(Student's t, pooled s = 0.0671018)
Indicating that we've recovered the regression from enabling semaphores
on this saturated setup, with a hint towards an overall improvement.
Very similar, but of smaller magnitude, results are observed on both
Skylake(gt2) and Kabylake(gt4). This may be due to the reduced impact of
bus-cycles, where we see a 50% hit on Broxton, it is only 10% on the big
core, in this particular test.
One observation to make here is that for a greedy client trying to
maximise its own throughput, using semaphores is the right choice. It is
only the holistic system-wide view that semaphores of one client
impacts another and reduces the overall throughput where we would choose
to disable semaphores.
The most noticeable negactive impact this has is on the no-op
microbenchmarks, which are also very notable for having no cpu bus load.
In particular, this increases the runtime and energy consumption of
gem_exec_whisper.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190504070707.30902-1-chris@chris-wilson.co.uk
2019-05-04 14:07:07 +07:00
|
|
|
|
2019-09-23 18:00:55 +07:00
|
|
|
engine->emit_fini_breadcrumb(request,
|
|
|
|
request->ring->vaddr + request->postfix);
|
2019-03-01 05:06:39 +07:00
|
|
|
|
2019-09-23 18:00:55 +07:00
|
|
|
trace_i915_request_execute(request);
|
|
|
|
engine->serial++;
|
|
|
|
result = true;
|
2019-06-14 23:46:06 +07:00
|
|
|
|
2019-09-23 18:00:55 +07:00
|
|
|
xfer: /* We may be recursing from the signal callback of another i915 fence */
|
|
|
|
spin_lock_nested(&request->lock, SINGLE_DEPTH_NESTING);
|
|
|
|
|
|
|
|
if (!test_and_set_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags))
|
|
|
|
list_move_tail(&request->sched.link, &engine->active.requests);
|
2019-03-01 05:06:39 +07:00
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &request->fence.flags) &&
|
drm/i915: Seal races between async GPU cancellation, retirement and signaling
Currently there is an underlying assumption that i915_request_unsubmit()
is synchronous wrt the GPU -- that is the request is no longer in flight
as we remove it. In the near future that may change, and this may upset
our signaling as we can process an interrupt for that request while it
is no longer in flight.
CPU0 CPU1
intel_engine_breadcrumbs_irq
(queue request completion)
i915_request_cancel_signaling
... ...
i915_request_enable_signaling
dma_fence_signal
Hence in the time it took us to drop the lock to signal the request, a
preemption event may have occurred and re-queued the request. In the
process, that request would have seen I915_FENCE_FLAG_SIGNAL clear and
so reused the rq->signal_link that was in use on CPU0, leading to bad
pointer chasing in intel_engine_breadcrumbs_irq.
A related issue was that if someone started listening for a signal on a
completed but no longer in-flight request, we missed the opportunity to
immediately signal that request.
Furthermore, as intel_contexts may be immediately released during
request retirement, in order to be entirely sure that
intel_engine_breadcrumbs_irq may no longer dereference the intel_context
(ce->signals and ce->signal_link), we must wait for irq spinlock.
In order to prevent the race, we use a bit in the fence.flags to signal
the transfer onto the signal list inside intel_engine_breadcrumbs_irq.
For simplicity, we use the DMA_FENCE_FLAG_SIGNALED_BIT as it then
quickly signals to any outside observer that the fence is indeed signaled.
v2: Sketch out potential dma-fence API for manual signaling
v3: And the test_and_set_bit()
Fixes: 52c0fdb25c7c ("drm/i915: Replace global breadcrumbs with per-context interrupt tracking")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190508112452.18942-1-chris@chris-wilson.co.uk
2019-05-08 18:24:52 +07:00
|
|
|
!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &request->fence.flags) &&
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
!i915_request_enable_breadcrumb(request))
|
2019-12-17 16:56:41 +07:00
|
|
|
intel_engine_signal_breadcrumbs(engine);
|
2019-03-01 05:06:39 +07:00
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
__notify_execute_cb(request);
|
|
|
|
|
2016-10-28 19:58:57 +07:00
|
|
|
spin_unlock(&request->lock);
|
|
|
|
|
2019-09-23 18:00:55 +07:00
|
|
|
return result;
|
2016-11-15 03:40:59 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void i915_request_submit(struct i915_request *request)
|
2016-11-15 03:40:59 +07:00
|
|
|
{
|
|
|
|
struct intel_engine_cs *engine = request->engine;
|
|
|
|
unsigned long flags;
|
2016-11-15 03:40:58 +07:00
|
|
|
|
2016-11-15 03:40:59 +07:00
|
|
|
/* Will be called from irq-context when using foreign fences. */
|
2019-06-14 23:46:06 +07:00
|
|
|
spin_lock_irqsave(&engine->active.lock, flags);
|
2016-11-15 03:40:59 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
__i915_request_submit(request);
|
2016-11-15 03:40:59 +07:00
|
|
|
|
2019-06-14 23:46:06 +07:00
|
|
|
spin_unlock_irqrestore(&engine->active.lock, flags);
|
2016-11-15 03:40:59 +07:00
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void __i915_request_unsubmit(struct i915_request *request)
|
2016-11-15 03:40:59 +07:00
|
|
|
{
|
2017-02-23 14:44:17 +07:00
|
|
|
struct intel_engine_cs *engine = request->engine;
|
2016-11-15 03:40:59 +07:00
|
|
|
|
2019-12-13 22:51:52 +07:00
|
|
|
RQ_TRACE(request, "\n");
|
2018-03-15 20:14:50 +07:00
|
|
|
|
2017-03-02 18:51:30 +07:00
|
|
|
GEM_BUG_ON(!irqs_disabled());
|
2019-06-14 23:46:06 +07:00
|
|
|
lockdep_assert_held(&engine->active.lock);
|
2016-11-25 20:17:17 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* Only unwind in reverse order, required so that the per-context list
|
2017-02-23 14:44:17 +07:00
|
|
|
* is kept in seqno/ring order.
|
|
|
|
*/
|
2016-10-28 19:58:58 +07:00
|
|
|
|
2017-02-23 14:44:17 +07:00
|
|
|
/* We may be recursing from the signal callback of another i915 fence */
|
|
|
|
spin_lock_nested(&request->lock, SINGLE_DEPTH_NESTING);
|
2019-03-01 05:06:39 +07:00
|
|
|
|
2017-02-23 14:44:17 +07:00
|
|
|
if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &request->fence.flags))
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
i915_request_cancel_breadcrumb(request);
|
2019-03-01 05:06:39 +07:00
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
GEM_BUG_ON(!test_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags));
|
|
|
|
clear_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags);
|
2019-03-01 05:06:39 +07:00
|
|
|
|
2017-02-23 14:44:17 +07:00
|
|
|
spin_unlock(&request->lock);
|
|
|
|
|
2019-05-15 20:00:48 +07:00
|
|
|
/* We've already spun, don't charge on resubmitting. */
|
|
|
|
if (request->sched.semaphores && i915_request_started(request)) {
|
|
|
|
request->sched.attr.priority |= I915_PRIORITY_NOSEMAPHORE;
|
|
|
|
request->sched.semaphores = 0;
|
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* We don't need to wake_up any waiters on request->execute, they
|
2017-02-23 14:44:17 +07:00
|
|
|
* will get woken by any other event or us re-adding this request
|
2018-02-21 16:56:36 +07:00
|
|
|
* to the engine timeline (__i915_request_submit()). The waiters
|
2017-02-23 14:44:17 +07:00
|
|
|
* should be quite adapt at finding that the request now has a new
|
|
|
|
* global_seqno to the one they went to sleep on.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
void i915_request_unsubmit(struct i915_request *request)
|
2017-02-23 14:44:17 +07:00
|
|
|
{
|
|
|
|
struct intel_engine_cs *engine = request->engine;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
/* Will be called from irq-context when using foreign fences. */
|
2019-06-14 23:46:06 +07:00
|
|
|
spin_lock_irqsave(&engine->active.lock, flags);
|
2017-02-23 14:44:17 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
__i915_request_unsubmit(request);
|
2017-02-23 14:44:17 +07:00
|
|
|
|
2019-06-14 23:46:06 +07:00
|
|
|
spin_unlock_irqrestore(&engine->active.lock, flags);
|
2016-09-09 20:11:54 +07:00
|
|
|
}
|
|
|
|
|
2016-11-15 03:40:58 +07:00
|
|
|
static int __i915_sw_fence_call
|
2016-11-15 03:40:59 +07:00
|
|
|
submit_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state)
|
2016-11-15 03:40:58 +07:00
|
|
|
{
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_request *request =
|
2016-11-25 20:17:17 +07:00
|
|
|
container_of(fence, typeof(*request), submit);
|
|
|
|
|
|
|
|
switch (state) {
|
|
|
|
case FENCE_COMPLETE:
|
2018-02-21 16:56:36 +07:00
|
|
|
trace_i915_request_submit(request);
|
2019-08-18 06:25:11 +07:00
|
|
|
|
|
|
|
if (unlikely(fence->error))
|
|
|
|
i915_request_skip(request, fence->error);
|
|
|
|
|
drm/i915: Use rcu instead of stop_machine in set_wedged
stop_machine is not really a locking primitive we should use, except
when the hw folks tell us the hw is broken and that's the only way to
work around it.
This patch tries to address the locking abuse of stop_machine() from
commit 20e4933c478a1ca694b38fa4ac44d99e659941f5
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date: Tue Nov 22 14:41:21 2016 +0000
drm/i915: Stop the machine as we install the wedged submit_request handler
Chris said parts of the reasons for going with stop_machine() was that
it's no overhead for the fast-path. But these callbacks use irqsave
spinlocks and do a bunch of MMIO, and rcu_read_lock is _real_ fast.
To stay as close as possible to the stop_machine semantics we first
update all the submit function pointers to the nop handler, then call
synchronize_rcu() to make sure no new requests can be submitted. This
should give us exactly the huge barrier we want.
I pondered whether we should annotate engine->submit_request as __rcu
and use rcu_assign_pointer and rcu_dereference on it. But the reason
behind those is to make sure the compiler/cpu barriers are there for
when you have an actual data structure you point at, to make sure all
the writes are seen correctly on the read side. But we just have a
function pointer, and .text isn't changed, so no need for these
barriers and hence no need for annotations.
Unfortunately there's a complication with the call to
intel_engine_init_global_seqno:
- Without stop_machine we must hold the corresponding spinlock.
- Without stop_machine we must ensure that all requests are marked as
having failed with dma_fence_set_error() before we call it. That
means we need to split the nop request submission into two phases,
both synchronized with rcu:
1. Only stop submitting the requests to hw and mark them as failed.
2. After all pending requests in the scheduler/ring are suitably
marked up as failed and we can force complete them all, also force
complete by calling intel_engine_init_global_seqno().
This should fix the followwing lockdep splat:
======================================================
WARNING: possible circular locking dependency detected
4.14.0-rc3-CI-CI_DRM_3179+ #1 Tainted: G U
------------------------------------------------------
kworker/3:4/562 is trying to acquire lock:
(cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8113d4bc>] stop_machine+0x1c/0x40
but task is already holding lock:
(&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #6 (&dev->struct_mutex){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__mutex_lock+0x86/0x9b0
mutex_lock_interruptible_nested+0x1b/0x20
i915_mutex_lock_interruptible+0x51/0x130 [i915]
i915_gem_fault+0x209/0x650 [i915]
__do_fault+0x1e/0x80
__handle_mm_fault+0xa08/0xed0
handle_mm_fault+0x156/0x300
__do_page_fault+0x2c5/0x570
do_page_fault+0x28/0x250
page_fault+0x22/0x30
-> #5 (&mm->mmap_sem){++++}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__might_fault+0x68/0x90
_copy_to_user+0x23/0x70
filldir+0xa5/0x120
dcache_readdir+0xf9/0x170
iterate_dir+0x69/0x1a0
SyS_getdents+0xa5/0x140
entry_SYSCALL_64_fastpath+0x1c/0xb1
-> #4 (&sb->s_type->i_mutex_key#5){++++}:
down_write+0x3b/0x70
handle_create+0xcb/0x1e0
devtmpfsd+0x139/0x180
kthread+0x152/0x190
ret_from_fork+0x27/0x40
-> #3 ((complete)&req.done){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
wait_for_common+0x58/0x210
wait_for_completion+0x1d/0x20
devtmpfs_create_node+0x13d/0x160
device_add+0x5eb/0x620
device_create_groups_vargs+0xe0/0xf0
device_create+0x3a/0x40
msr_device_create+0x2b/0x40
cpuhp_invoke_callback+0xc9/0xbf0
cpuhp_thread_fun+0x17b/0x240
smpboot_thread_fn+0x18a/0x280
kthread+0x152/0x190
ret_from_fork+0x27/0x40
-> #2 (cpuhp_state-up){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
cpuhp_issue_call+0x133/0x1c0
__cpuhp_setup_state_cpuslocked+0x139/0x2a0
__cpuhp_setup_state+0x46/0x60
page_writeback_init+0x43/0x67
pagecache_init+0x3d/0x42
start_kernel+0x3a8/0x3fc
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x6d/0x70
verify_cpu+0x0/0xfb
-> #1 (cpuhp_state_mutex){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__mutex_lock+0x86/0x9b0
mutex_lock_nested+0x1b/0x20
__cpuhp_setup_state_cpuslocked+0x53/0x2a0
__cpuhp_setup_state+0x46/0x60
page_alloc_init+0x28/0x30
start_kernel+0x145/0x3fc
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x6d/0x70
verify_cpu+0x0/0xfb
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
check_prev_add+0x430/0x840
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
cpus_read_lock+0x3d/0xb0
stop_machine+0x1c/0x40
i915_gem_set_wedged+0x1a/0x20 [i915]
i915_reset+0xb9/0x230 [i915]
i915_reset_device+0x1f6/0x260 [i915]
i915_handle_error+0x2d8/0x430 [i915]
hangcheck_declare_hang+0xd3/0xf0 [i915]
i915_hangcheck_elapsed+0x262/0x2d0 [i915]
process_one_work+0x233/0x660
worker_thread+0x4e/0x3b0
kthread+0x152/0x190
ret_from_fork+0x27/0x40
other info that might help us debug this:
Chain exists of:
cpu_hotplug_lock.rw_sem --> &mm->mmap_sem --> &dev->struct_mutex
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&dev->struct_mutex);
lock(&mm->mmap_sem);
lock(&dev->struct_mutex);
lock(cpu_hotplug_lock.rw_sem);
*** DEADLOCK ***
3 locks held by kworker/3:4/562:
#0: ("events_long"){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
#1: ((&(&i915->gpu_error.hangcheck_work)->work)){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
#2: (&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
stack backtrace:
CPU: 3 PID: 562 Comm: kworker/3:4 Tainted: G U 4.14.0-rc3-CI-CI_DRM_3179+ #1
Hardware name: /NUC7i5BNB, BIOS BNKBL357.86A.0048.2017.0704.1415 07/04/2017
Workqueue: events_long i915_hangcheck_elapsed [i915]
Call Trace:
dump_stack+0x68/0x9f
print_circular_bug+0x235/0x3c0
? lockdep_init_map_crosslock+0x20/0x20
check_prev_add+0x430/0x840
? irq_work_queue+0x86/0xe0
? wake_up_klogd+0x53/0x70
__lock_acquire+0x1420/0x15e0
? __lock_acquire+0x1420/0x15e0
? lockdep_init_map_crosslock+0x20/0x20
lock_acquire+0xb0/0x200
? stop_machine+0x1c/0x40
? i915_gem_object_truncate+0x50/0x50 [i915]
cpus_read_lock+0x3d/0xb0
? stop_machine+0x1c/0x40
stop_machine+0x1c/0x40
i915_gem_set_wedged+0x1a/0x20 [i915]
i915_reset+0xb9/0x230 [i915]
i915_reset_device+0x1f6/0x260 [i915]
? gen8_gt_irq_ack+0x170/0x170 [i915]
? work_on_cpu_safe+0x60/0x60
i915_handle_error+0x2d8/0x430 [i915]
? vsnprintf+0xd1/0x4b0
? scnprintf+0x3a/0x70
hangcheck_declare_hang+0xd3/0xf0 [i915]
? intel_runtime_pm_put+0x56/0xa0 [i915]
i915_hangcheck_elapsed+0x262/0x2d0 [i915]
process_one_work+0x233/0x660
worker_thread+0x4e/0x3b0
kthread+0x152/0x190
? process_one_work+0x660/0x660
? kthread_create_on_node+0x40/0x40
ret_from_fork+0x27/0x40
Setting dangerous option reset - tainting kernel
i915 0000:00:02.0: Resetting chip after gpu hang
Setting dangerous option reset - tainting kernel
i915 0000:00:02.0: Resetting chip after gpu hang
v2: Have 1 global synchronize_rcu() barrier across all engines, and
improve commit message.
v3: We need to protect the seqno update with the timeline spinlock (in
set_wedged) to avoid racing with other updates of the seqno, like we
already do in nop_submit_request (Chris).
v4: Use two-phase sequence to plug the race Chris spotted where we can
complete requests before they're marked up with -EIO.
v5: Review from Chris:
- simplify nop_submit_request.
- Add comment to rcu_read_lock section.
- Align comments with the new style.
v6: Remove unused variable to appease CI.
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102886
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103096
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marta Lofstedt <marta.lofstedt@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171011091019.1425-1-daniel.vetter@ffwll.ch
2017-10-11 16:10:19 +07:00
|
|
|
/*
|
2018-02-21 16:56:36 +07:00
|
|
|
* We need to serialize use of the submit_request() callback
|
|
|
|
* with its hotplugging performed during an emergency
|
|
|
|
* i915_gem_set_wedged(). We use the RCU mechanism to mark the
|
|
|
|
* critical section in order to force i915_gem_set_wedged() to
|
|
|
|
* wait until the submit_request() is completed before
|
|
|
|
* proceeding.
|
drm/i915: Use rcu instead of stop_machine in set_wedged
stop_machine is not really a locking primitive we should use, except
when the hw folks tell us the hw is broken and that's the only way to
work around it.
This patch tries to address the locking abuse of stop_machine() from
commit 20e4933c478a1ca694b38fa4ac44d99e659941f5
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date: Tue Nov 22 14:41:21 2016 +0000
drm/i915: Stop the machine as we install the wedged submit_request handler
Chris said parts of the reasons for going with stop_machine() was that
it's no overhead for the fast-path. But these callbacks use irqsave
spinlocks and do a bunch of MMIO, and rcu_read_lock is _real_ fast.
To stay as close as possible to the stop_machine semantics we first
update all the submit function pointers to the nop handler, then call
synchronize_rcu() to make sure no new requests can be submitted. This
should give us exactly the huge barrier we want.
I pondered whether we should annotate engine->submit_request as __rcu
and use rcu_assign_pointer and rcu_dereference on it. But the reason
behind those is to make sure the compiler/cpu barriers are there for
when you have an actual data structure you point at, to make sure all
the writes are seen correctly on the read side. But we just have a
function pointer, and .text isn't changed, so no need for these
barriers and hence no need for annotations.
Unfortunately there's a complication with the call to
intel_engine_init_global_seqno:
- Without stop_machine we must hold the corresponding spinlock.
- Without stop_machine we must ensure that all requests are marked as
having failed with dma_fence_set_error() before we call it. That
means we need to split the nop request submission into two phases,
both synchronized with rcu:
1. Only stop submitting the requests to hw and mark them as failed.
2. After all pending requests in the scheduler/ring are suitably
marked up as failed and we can force complete them all, also force
complete by calling intel_engine_init_global_seqno().
This should fix the followwing lockdep splat:
======================================================
WARNING: possible circular locking dependency detected
4.14.0-rc3-CI-CI_DRM_3179+ #1 Tainted: G U
------------------------------------------------------
kworker/3:4/562 is trying to acquire lock:
(cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8113d4bc>] stop_machine+0x1c/0x40
but task is already holding lock:
(&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #6 (&dev->struct_mutex){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__mutex_lock+0x86/0x9b0
mutex_lock_interruptible_nested+0x1b/0x20
i915_mutex_lock_interruptible+0x51/0x130 [i915]
i915_gem_fault+0x209/0x650 [i915]
__do_fault+0x1e/0x80
__handle_mm_fault+0xa08/0xed0
handle_mm_fault+0x156/0x300
__do_page_fault+0x2c5/0x570
do_page_fault+0x28/0x250
page_fault+0x22/0x30
-> #5 (&mm->mmap_sem){++++}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__might_fault+0x68/0x90
_copy_to_user+0x23/0x70
filldir+0xa5/0x120
dcache_readdir+0xf9/0x170
iterate_dir+0x69/0x1a0
SyS_getdents+0xa5/0x140
entry_SYSCALL_64_fastpath+0x1c/0xb1
-> #4 (&sb->s_type->i_mutex_key#5){++++}:
down_write+0x3b/0x70
handle_create+0xcb/0x1e0
devtmpfsd+0x139/0x180
kthread+0x152/0x190
ret_from_fork+0x27/0x40
-> #3 ((complete)&req.done){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
wait_for_common+0x58/0x210
wait_for_completion+0x1d/0x20
devtmpfs_create_node+0x13d/0x160
device_add+0x5eb/0x620
device_create_groups_vargs+0xe0/0xf0
device_create+0x3a/0x40
msr_device_create+0x2b/0x40
cpuhp_invoke_callback+0xc9/0xbf0
cpuhp_thread_fun+0x17b/0x240
smpboot_thread_fn+0x18a/0x280
kthread+0x152/0x190
ret_from_fork+0x27/0x40
-> #2 (cpuhp_state-up){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
cpuhp_issue_call+0x133/0x1c0
__cpuhp_setup_state_cpuslocked+0x139/0x2a0
__cpuhp_setup_state+0x46/0x60
page_writeback_init+0x43/0x67
pagecache_init+0x3d/0x42
start_kernel+0x3a8/0x3fc
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x6d/0x70
verify_cpu+0x0/0xfb
-> #1 (cpuhp_state_mutex){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__mutex_lock+0x86/0x9b0
mutex_lock_nested+0x1b/0x20
__cpuhp_setup_state_cpuslocked+0x53/0x2a0
__cpuhp_setup_state+0x46/0x60
page_alloc_init+0x28/0x30
start_kernel+0x145/0x3fc
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x6d/0x70
verify_cpu+0x0/0xfb
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
check_prev_add+0x430/0x840
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
cpus_read_lock+0x3d/0xb0
stop_machine+0x1c/0x40
i915_gem_set_wedged+0x1a/0x20 [i915]
i915_reset+0xb9/0x230 [i915]
i915_reset_device+0x1f6/0x260 [i915]
i915_handle_error+0x2d8/0x430 [i915]
hangcheck_declare_hang+0xd3/0xf0 [i915]
i915_hangcheck_elapsed+0x262/0x2d0 [i915]
process_one_work+0x233/0x660
worker_thread+0x4e/0x3b0
kthread+0x152/0x190
ret_from_fork+0x27/0x40
other info that might help us debug this:
Chain exists of:
cpu_hotplug_lock.rw_sem --> &mm->mmap_sem --> &dev->struct_mutex
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&dev->struct_mutex);
lock(&mm->mmap_sem);
lock(&dev->struct_mutex);
lock(cpu_hotplug_lock.rw_sem);
*** DEADLOCK ***
3 locks held by kworker/3:4/562:
#0: ("events_long"){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
#1: ((&(&i915->gpu_error.hangcheck_work)->work)){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
#2: (&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
stack backtrace:
CPU: 3 PID: 562 Comm: kworker/3:4 Tainted: G U 4.14.0-rc3-CI-CI_DRM_3179+ #1
Hardware name: /NUC7i5BNB, BIOS BNKBL357.86A.0048.2017.0704.1415 07/04/2017
Workqueue: events_long i915_hangcheck_elapsed [i915]
Call Trace:
dump_stack+0x68/0x9f
print_circular_bug+0x235/0x3c0
? lockdep_init_map_crosslock+0x20/0x20
check_prev_add+0x430/0x840
? irq_work_queue+0x86/0xe0
? wake_up_klogd+0x53/0x70
__lock_acquire+0x1420/0x15e0
? __lock_acquire+0x1420/0x15e0
? lockdep_init_map_crosslock+0x20/0x20
lock_acquire+0xb0/0x200
? stop_machine+0x1c/0x40
? i915_gem_object_truncate+0x50/0x50 [i915]
cpus_read_lock+0x3d/0xb0
? stop_machine+0x1c/0x40
stop_machine+0x1c/0x40
i915_gem_set_wedged+0x1a/0x20 [i915]
i915_reset+0xb9/0x230 [i915]
i915_reset_device+0x1f6/0x260 [i915]
? gen8_gt_irq_ack+0x170/0x170 [i915]
? work_on_cpu_safe+0x60/0x60
i915_handle_error+0x2d8/0x430 [i915]
? vsnprintf+0xd1/0x4b0
? scnprintf+0x3a/0x70
hangcheck_declare_hang+0xd3/0xf0 [i915]
? intel_runtime_pm_put+0x56/0xa0 [i915]
i915_hangcheck_elapsed+0x262/0x2d0 [i915]
process_one_work+0x233/0x660
worker_thread+0x4e/0x3b0
kthread+0x152/0x190
? process_one_work+0x660/0x660
? kthread_create_on_node+0x40/0x40
ret_from_fork+0x27/0x40
Setting dangerous option reset - tainting kernel
i915 0000:00:02.0: Resetting chip after gpu hang
Setting dangerous option reset - tainting kernel
i915 0000:00:02.0: Resetting chip after gpu hang
v2: Have 1 global synchronize_rcu() barrier across all engines, and
improve commit message.
v3: We need to protect the seqno update with the timeline spinlock (in
set_wedged) to avoid racing with other updates of the seqno, like we
already do in nop_submit_request (Chris).
v4: Use two-phase sequence to plug the race Chris spotted where we can
complete requests before they're marked up with -EIO.
v5: Review from Chris:
- simplify nop_submit_request.
- Add comment to rcu_read_lock section.
- Align comments with the new style.
v6: Remove unused variable to appease CI.
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102886
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103096
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marta Lofstedt <marta.lofstedt@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171011091019.1425-1-daniel.vetter@ffwll.ch
2017-10-11 16:10:19 +07:00
|
|
|
*/
|
|
|
|
rcu_read_lock();
|
2016-11-15 03:40:59 +07:00
|
|
|
request->engine->submit_request(request);
|
drm/i915: Use rcu instead of stop_machine in set_wedged
stop_machine is not really a locking primitive we should use, except
when the hw folks tell us the hw is broken and that's the only way to
work around it.
This patch tries to address the locking abuse of stop_machine() from
commit 20e4933c478a1ca694b38fa4ac44d99e659941f5
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date: Tue Nov 22 14:41:21 2016 +0000
drm/i915: Stop the machine as we install the wedged submit_request handler
Chris said parts of the reasons for going with stop_machine() was that
it's no overhead for the fast-path. But these callbacks use irqsave
spinlocks and do a bunch of MMIO, and rcu_read_lock is _real_ fast.
To stay as close as possible to the stop_machine semantics we first
update all the submit function pointers to the nop handler, then call
synchronize_rcu() to make sure no new requests can be submitted. This
should give us exactly the huge barrier we want.
I pondered whether we should annotate engine->submit_request as __rcu
and use rcu_assign_pointer and rcu_dereference on it. But the reason
behind those is to make sure the compiler/cpu barriers are there for
when you have an actual data structure you point at, to make sure all
the writes are seen correctly on the read side. But we just have a
function pointer, and .text isn't changed, so no need for these
barriers and hence no need for annotations.
Unfortunately there's a complication with the call to
intel_engine_init_global_seqno:
- Without stop_machine we must hold the corresponding spinlock.
- Without stop_machine we must ensure that all requests are marked as
having failed with dma_fence_set_error() before we call it. That
means we need to split the nop request submission into two phases,
both synchronized with rcu:
1. Only stop submitting the requests to hw and mark them as failed.
2. After all pending requests in the scheduler/ring are suitably
marked up as failed and we can force complete them all, also force
complete by calling intel_engine_init_global_seqno().
This should fix the followwing lockdep splat:
======================================================
WARNING: possible circular locking dependency detected
4.14.0-rc3-CI-CI_DRM_3179+ #1 Tainted: G U
------------------------------------------------------
kworker/3:4/562 is trying to acquire lock:
(cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8113d4bc>] stop_machine+0x1c/0x40
but task is already holding lock:
(&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #6 (&dev->struct_mutex){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__mutex_lock+0x86/0x9b0
mutex_lock_interruptible_nested+0x1b/0x20
i915_mutex_lock_interruptible+0x51/0x130 [i915]
i915_gem_fault+0x209/0x650 [i915]
__do_fault+0x1e/0x80
__handle_mm_fault+0xa08/0xed0
handle_mm_fault+0x156/0x300
__do_page_fault+0x2c5/0x570
do_page_fault+0x28/0x250
page_fault+0x22/0x30
-> #5 (&mm->mmap_sem){++++}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__might_fault+0x68/0x90
_copy_to_user+0x23/0x70
filldir+0xa5/0x120
dcache_readdir+0xf9/0x170
iterate_dir+0x69/0x1a0
SyS_getdents+0xa5/0x140
entry_SYSCALL_64_fastpath+0x1c/0xb1
-> #4 (&sb->s_type->i_mutex_key#5){++++}:
down_write+0x3b/0x70
handle_create+0xcb/0x1e0
devtmpfsd+0x139/0x180
kthread+0x152/0x190
ret_from_fork+0x27/0x40
-> #3 ((complete)&req.done){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
wait_for_common+0x58/0x210
wait_for_completion+0x1d/0x20
devtmpfs_create_node+0x13d/0x160
device_add+0x5eb/0x620
device_create_groups_vargs+0xe0/0xf0
device_create+0x3a/0x40
msr_device_create+0x2b/0x40
cpuhp_invoke_callback+0xc9/0xbf0
cpuhp_thread_fun+0x17b/0x240
smpboot_thread_fn+0x18a/0x280
kthread+0x152/0x190
ret_from_fork+0x27/0x40
-> #2 (cpuhp_state-up){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
cpuhp_issue_call+0x133/0x1c0
__cpuhp_setup_state_cpuslocked+0x139/0x2a0
__cpuhp_setup_state+0x46/0x60
page_writeback_init+0x43/0x67
pagecache_init+0x3d/0x42
start_kernel+0x3a8/0x3fc
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x6d/0x70
verify_cpu+0x0/0xfb
-> #1 (cpuhp_state_mutex){+.+.}:
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
__mutex_lock+0x86/0x9b0
mutex_lock_nested+0x1b/0x20
__cpuhp_setup_state_cpuslocked+0x53/0x2a0
__cpuhp_setup_state+0x46/0x60
page_alloc_init+0x28/0x30
start_kernel+0x145/0x3fc
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x6d/0x70
verify_cpu+0x0/0xfb
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
check_prev_add+0x430/0x840
__lock_acquire+0x1420/0x15e0
lock_acquire+0xb0/0x200
cpus_read_lock+0x3d/0xb0
stop_machine+0x1c/0x40
i915_gem_set_wedged+0x1a/0x20 [i915]
i915_reset+0xb9/0x230 [i915]
i915_reset_device+0x1f6/0x260 [i915]
i915_handle_error+0x2d8/0x430 [i915]
hangcheck_declare_hang+0xd3/0xf0 [i915]
i915_hangcheck_elapsed+0x262/0x2d0 [i915]
process_one_work+0x233/0x660
worker_thread+0x4e/0x3b0
kthread+0x152/0x190
ret_from_fork+0x27/0x40
other info that might help us debug this:
Chain exists of:
cpu_hotplug_lock.rw_sem --> &mm->mmap_sem --> &dev->struct_mutex
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&dev->struct_mutex);
lock(&mm->mmap_sem);
lock(&dev->struct_mutex);
lock(cpu_hotplug_lock.rw_sem);
*** DEADLOCK ***
3 locks held by kworker/3:4/562:
#0: ("events_long"){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
#1: ((&(&i915->gpu_error.hangcheck_work)->work)){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
#2: (&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
stack backtrace:
CPU: 3 PID: 562 Comm: kworker/3:4 Tainted: G U 4.14.0-rc3-CI-CI_DRM_3179+ #1
Hardware name: /NUC7i5BNB, BIOS BNKBL357.86A.0048.2017.0704.1415 07/04/2017
Workqueue: events_long i915_hangcheck_elapsed [i915]
Call Trace:
dump_stack+0x68/0x9f
print_circular_bug+0x235/0x3c0
? lockdep_init_map_crosslock+0x20/0x20
check_prev_add+0x430/0x840
? irq_work_queue+0x86/0xe0
? wake_up_klogd+0x53/0x70
__lock_acquire+0x1420/0x15e0
? __lock_acquire+0x1420/0x15e0
? lockdep_init_map_crosslock+0x20/0x20
lock_acquire+0xb0/0x200
? stop_machine+0x1c/0x40
? i915_gem_object_truncate+0x50/0x50 [i915]
cpus_read_lock+0x3d/0xb0
? stop_machine+0x1c/0x40
stop_machine+0x1c/0x40
i915_gem_set_wedged+0x1a/0x20 [i915]
i915_reset+0xb9/0x230 [i915]
i915_reset_device+0x1f6/0x260 [i915]
? gen8_gt_irq_ack+0x170/0x170 [i915]
? work_on_cpu_safe+0x60/0x60
i915_handle_error+0x2d8/0x430 [i915]
? vsnprintf+0xd1/0x4b0
? scnprintf+0x3a/0x70
hangcheck_declare_hang+0xd3/0xf0 [i915]
? intel_runtime_pm_put+0x56/0xa0 [i915]
i915_hangcheck_elapsed+0x262/0x2d0 [i915]
process_one_work+0x233/0x660
worker_thread+0x4e/0x3b0
kthread+0x152/0x190
? process_one_work+0x660/0x660
? kthread_create_on_node+0x40/0x40
ret_from_fork+0x27/0x40
Setting dangerous option reset - tainting kernel
i915 0000:00:02.0: Resetting chip after gpu hang
Setting dangerous option reset - tainting kernel
i915 0000:00:02.0: Resetting chip after gpu hang
v2: Have 1 global synchronize_rcu() barrier across all engines, and
improve commit message.
v3: We need to protect the seqno update with the timeline spinlock (in
set_wedged) to avoid racing with other updates of the seqno, like we
already do in nop_submit_request (Chris).
v4: Use two-phase sequence to plug the race Chris spotted where we can
complete requests before they're marked up with -EIO.
v5: Review from Chris:
- simplify nop_submit_request.
- Add comment to rcu_read_lock section.
- Align comments with the new style.
v6: Remove unused variable to appease CI.
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102886
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103096
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Mika Kuoppala <mika.kuoppala@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marta Lofstedt <marta.lofstedt@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20171011091019.1425-1-daniel.vetter@ffwll.ch
2017-10-11 16:10:19 +07:00
|
|
|
rcu_read_unlock();
|
2016-11-25 20:17:17 +07:00
|
|
|
break;
|
|
|
|
|
|
|
|
case FENCE_FREE:
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_request_put(request);
|
2016-11-25 20:17:17 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2016-11-15 03:40:58 +07:00
|
|
|
return NOTIFY_DONE;
|
|
|
|
}
|
|
|
|
|
drm/i915: Bump ready tasks ahead of busywaits
Consider two tasks that are running in parallel on a pair of engines
(vcs0, vcs1), but then must complete on a shared engine (rcs0). To
maximise throughput, we want to run the first ready task on rcs0 (i.e.
the first task that completes on either of vcs0 or vcs1). When using
semaphores, however, we will instead queue onto rcs in submission order.
To resolve this incorrect ordering, we want to re-evaluate the priority
queue when each of the request is ready. Normally this happens because
we only insert into the priority queue requests that are ready, but with
semaphores we are inserting ahead of their readiness and to compensate
we penalize those tasks with reduced priority (so that tasks that do not
need to busywait should naturally be run first). However, given a series
of tasks that each use semaphores, the queue degrades into submission
fifo rather than readiness fifo, and so to counter this we give a small
boost to semaphore users as their dependent tasks are completed (and so
we no longer require any busywait prior to running the user task as they
are then ready themselves).
v2: Fixup irqsave for schedule_lock (Tvrtko)
Testcase: igt/gem_exec_schedule/semaphore-codependency
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190409152922.23894-1-chris@chris-wilson.co.uk
2019-04-09 22:29:22 +07:00
|
|
|
static int __i915_sw_fence_call
|
|
|
|
semaphore_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state)
|
|
|
|
{
|
|
|
|
struct i915_request *request =
|
|
|
|
container_of(fence, typeof(*request), semaphore);
|
|
|
|
|
|
|
|
switch (state) {
|
|
|
|
case FENCE_COMPLETE:
|
drm/i915: Truly bump ready tasks ahead of busywaits
In commit b7404c7ecb38 ("drm/i915: Bump ready tasks ahead of
busywaits"), I tried cutting a corner in order to not install a signal
for each of our dependencies, and only listened to requests on which we
were intending to busywait. The compromise that was made was that
instead of then being able to promote the request with a full
NOSEMAPHORE like its non-busywaiting brethren, as we had not ensured we
had cleared the semaphore chain, we settled for only using the NEWCLIENT
boost. With an over saturated system with multiple NEWCLIENTS in flight
at any time, this was found to be an inadequate promotion and left us
with a much poorer scheduling order than prior to using semaphores.
The outcome of this patch, is that all requests have NOSEMAPHORE
priority when they have no dependencies and are ready to run and not
busywait, restoring the pre-semaphore ordering on saturated systems.
We can demonstrate the effect of poor scheduling order by oversaturating
the system using gem_wsim on a system with multiple vcs engines
(i.e running the same workloads across more clients than required for
peak throughput, e.g. media_load_balance_17i7.wsim -c4 -b context):
x v5.1 (normalized)
+ tip
* fix
+------------------------------------------------------------------------+
| x |
| x |
| x |
| x |
| %x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %#x |
| %#x |
| %#x |
| %#x |
| %#x |
| + %#xx |
| + %#xx |
| + %%#xx |
| + %%#xx |
| + %%#xx |
| + %%#xx |
| + %%##x |
| +++ %%##x |
| +++ %%##x |
| +++ %%##x |
| ++++ %%##x |
| ++++ %%##x |
| ++++ %%##xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ + %#O#xx |
| ++++ + %#O#xx |
| ++++++ + %#O#xx |
| ++++++++++ %OOOxxx|
| ++++++++++ + %#OOO#xx|
| + ++++++++++++ ++ +++++ + ++ @@OOOO#xx|
| |A_| |
||__________M_______A____________________| |
| |A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 0.99456 1.00628 0.999985 1.0001545 0.0024387139
+ 120 0.873021 1.00037 0.884134 0.90148752 0.039190862
Difference at 99.5% confidence
-0.098667 +/- 0.0110762
-9.86517% +/- 1.10745%
(Student's t, pooled s = 0.0277657)
% 120 0.990207 1.00165 0.9970265 0.99699748 0.0021024
Difference at 99.5% confidence
-0.003157 +/- 0.000908245
-0.315651% +/- 0.0908105%
(Student's t, pooled s = 0.00227678)
Fixes: b7404c7ecb38 ("drm/i915: Bump ready tasks ahead of busywaits")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190515130052.4475-2-chris@chris-wilson.co.uk
2019-05-15 20:00:49 +07:00
|
|
|
i915_schedule_bump_priority(request, I915_PRIORITY_NOSEMAPHORE);
|
drm/i915: Bump ready tasks ahead of busywaits
Consider two tasks that are running in parallel on a pair of engines
(vcs0, vcs1), but then must complete on a shared engine (rcs0). To
maximise throughput, we want to run the first ready task on rcs0 (i.e.
the first task that completes on either of vcs0 or vcs1). When using
semaphores, however, we will instead queue onto rcs in submission order.
To resolve this incorrect ordering, we want to re-evaluate the priority
queue when each of the request is ready. Normally this happens because
we only insert into the priority queue requests that are ready, but with
semaphores we are inserting ahead of their readiness and to compensate
we penalize those tasks with reduced priority (so that tasks that do not
need to busywait should naturally be run first). However, given a series
of tasks that each use semaphores, the queue degrades into submission
fifo rather than readiness fifo, and so to counter this we give a small
boost to semaphore users as their dependent tasks are completed (and so
we no longer require any busywait prior to running the user task as they
are then ready themselves).
v2: Fixup irqsave for schedule_lock (Tvrtko)
Testcase: igt/gem_exec_schedule/semaphore-codependency
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190409152922.23894-1-chris@chris-wilson.co.uk
2019-04-09 22:29:22 +07:00
|
|
|
break;
|
|
|
|
|
|
|
|
case FENCE_FREE:
|
|
|
|
i915_request_put(request);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return NOTIFY_DONE;
|
|
|
|
}
|
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
static void retire_requests(struct intel_timeline *tl)
|
2019-01-10 04:59:32 +07:00
|
|
|
{
|
|
|
|
struct i915_request *rq, *rn;
|
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
list_for_each_entry_safe(rq, rn, &tl->requests, link)
|
2019-06-14 23:46:05 +07:00
|
|
|
if (!i915_request_retire(rq))
|
2019-01-10 04:59:32 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
static noinline struct i915_request *
|
2019-08-16 03:57:09 +07:00
|
|
|
request_alloc_slow(struct intel_timeline *tl, gfp_t gfp)
|
2019-01-10 04:59:32 +07:00
|
|
|
{
|
|
|
|
struct i915_request *rq;
|
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
if (list_empty(&tl->requests))
|
2019-01-10 04:59:32 +07:00
|
|
|
goto out;
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
if (!gfpflags_allow_blocking(gfp))
|
|
|
|
goto out;
|
|
|
|
|
2019-06-14 23:46:05 +07:00
|
|
|
/* Move our oldest request to the slab-cache (if not in use!) */
|
2019-08-16 03:57:09 +07:00
|
|
|
rq = list_first_entry(&tl->requests, typeof(*rq), link);
|
2019-06-14 23:46:05 +07:00
|
|
|
i915_request_retire(rq);
|
|
|
|
|
|
|
|
rq = kmem_cache_alloc(global.slab_requests,
|
|
|
|
gfp | __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
|
|
|
|
if (rq)
|
|
|
|
return rq;
|
|
|
|
|
2019-01-10 04:59:32 +07:00
|
|
|
/* Ratelimit ourselves to prevent oom from malicious clients */
|
2019-08-16 03:57:09 +07:00
|
|
|
rq = list_last_entry(&tl->requests, typeof(*rq), link);
|
2019-01-10 04:59:32 +07:00
|
|
|
cond_synchronize_rcu(rq->rcustate);
|
|
|
|
|
|
|
|
/* Retire our old requests in the hope that we free some */
|
2019-08-16 03:57:09 +07:00
|
|
|
retire_requests(tl);
|
2019-01-10 04:59:32 +07:00
|
|
|
|
|
|
|
out:
|
2019-04-25 03:07:16 +07:00
|
|
|
return kmem_cache_alloc(global.slab_requests, gfp);
|
2019-01-10 04:59:32 +07:00
|
|
|
}
|
|
|
|
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
static void __i915_request_ctor(void *arg)
|
|
|
|
{
|
|
|
|
struct i915_request *rq = arg;
|
|
|
|
|
|
|
|
spin_lock_init(&rq->lock);
|
|
|
|
i915_sched_node_init(&rq->sched);
|
|
|
|
i915_sw_fence_init(&rq->submit, submit_notify);
|
|
|
|
i915_sw_fence_init(&rq->semaphore, semaphore_notify);
|
|
|
|
|
|
|
|
rq->file_priv = NULL;
|
|
|
|
rq->capture_list = NULL;
|
|
|
|
|
|
|
|
INIT_LIST_HEAD(&rq->execute_cb);
|
|
|
|
}
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
struct i915_request *
|
2019-04-25 03:07:16 +07:00
|
|
|
__i915_request_create(struct intel_context *ce, gfp_t gfp)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2019-08-10 01:25:18 +07:00
|
|
|
struct intel_timeline *tl = ce->timeline;
|
2019-03-02 00:08:59 +07:00
|
|
|
struct i915_request *rq;
|
|
|
|
u32 seqno;
|
2016-07-20 15:21:08 +07:00
|
|
|
int ret;
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
might_sleep_if(gfpflags_allow_blocking(gfp));
|
2016-10-28 19:58:56 +07:00
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
/* Check that the caller provided an already pinned context */
|
|
|
|
__intel_context_pin(ce);
|
2016-07-20 15:21:09 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* Beware: Dragons be flying overhead.
|
2016-08-09 15:23:34 +07:00
|
|
|
*
|
|
|
|
* We use RCU to look up requests in flight. The lookups may
|
|
|
|
* race with the request being allocated from the slab freelist.
|
|
|
|
* That is the request we are writing to here, may be in the process
|
drm/i915: Pull i915_gem_active into the i915_active family
Looking forward, we need to break the struct_mutex dependency on
i915_gem_active. In the meantime, external use of i915_gem_active is
quite beguiling, little do new users suspect that it implies a barrier
as each request it tracks must be ordered wrt the previous one. As one
of many, it can be used to track activity across multiple timelines, a
shared fence, which fits our unordered request submission much better. We
need to steer external users away from the singular, exclusive fence
imposed by i915_gem_active to i915_active instead. As part of that
process, we move i915_gem_active out of i915_request.c into
i915_active.c to start separating the two concepts, and rename it to
i915_active_request (both to tie it to the concept of tracking just one
request, and to give it a longer, less appealing name).
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190205130005.2807-5-chris@chris-wilson.co.uk
2019-02-05 20:00:05 +07:00
|
|
|
* of being read by __i915_active_request_get_rcu(). As such,
|
2016-08-09 15:23:34 +07:00
|
|
|
* we have to be very careful when overwriting the contents. During
|
|
|
|
* the RCU lookup, we change chase the request->engine pointer,
|
2016-10-28 19:58:49 +07:00
|
|
|
* read the request->global_seqno and increment the reference count.
|
2016-08-09 15:23:34 +07:00
|
|
|
*
|
|
|
|
* The reference count is incremented atomically. If it is zero,
|
|
|
|
* the lookup knows the request is unallocated and complete. Otherwise,
|
|
|
|
* it is either still in use, or has been reallocated and reset
|
2016-10-25 19:00:45 +07:00
|
|
|
* with dma_fence_init(). This increment is safe for release as we
|
|
|
|
* check that the request we have a reference to and matches the active
|
2016-08-09 15:23:34 +07:00
|
|
|
* request.
|
|
|
|
*
|
|
|
|
* Before we increment the refcount, we chase the request->engine
|
|
|
|
* pointer. We must not call kmem_cache_zalloc() or else we set
|
|
|
|
* that pointer to NULL and cause a crash during the lookup. If
|
|
|
|
* we see the request is completed (based on the value of the
|
|
|
|
* old engine and seqno), the lookup is complete and reports NULL.
|
|
|
|
* If we decide the request is not completed (new engine or seqno),
|
|
|
|
* then we grab a reference and double check that it is still the
|
|
|
|
* active request - which it won't be and restart the lookup.
|
|
|
|
*
|
|
|
|
* Do not use kmem_cache_zalloc() here!
|
|
|
|
*/
|
2019-02-28 17:20:33 +07:00
|
|
|
rq = kmem_cache_alloc(global.slab_requests,
|
2019-04-25 03:07:16 +07:00
|
|
|
gfp | __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
|
2018-02-21 16:56:36 +07:00
|
|
|
if (unlikely(!rq)) {
|
2019-08-16 03:57:09 +07:00
|
|
|
rq = request_alloc_slow(tl, gfp);
|
2018-02-21 16:56:36 +07:00
|
|
|
if (!rq) {
|
2017-12-13 01:06:52 +07:00
|
|
|
ret = -ENOMEM;
|
|
|
|
goto err_unreserve;
|
|
|
|
}
|
2016-10-28 19:58:56 +07:00
|
|
|
}
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-06-21 14:08:10 +07:00
|
|
|
ret = intel_timeline_get_seqno(tl, rq, &seqno);
|
2019-03-02 00:08:59 +07:00
|
|
|
if (ret)
|
|
|
|
goto err_free;
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
rq->i915 = ce->engine->i915;
|
2019-12-20 17:12:29 +07:00
|
|
|
rq->context = ce;
|
2019-04-25 03:07:16 +07:00
|
|
|
rq->engine = ce->engine;
|
2018-05-18 04:26:32 +07:00
|
|
|
rq->ring = ce->ring;
|
2019-10-14 03:30:12 +07:00
|
|
|
rq->execution_mask = ce->engine->mask;
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
rq->flags = 0;
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
|
2019-12-17 08:16:59 +07:00
|
|
|
RCU_INIT_POINTER(rq->timeline, tl);
|
|
|
|
RCU_INIT_POINTER(rq->hwsp_cacheline, tl->hwsp_cacheline);
|
2019-03-02 00:08:59 +07:00
|
|
|
rq->hwsp_seqno = tl->hwsp_seqno;
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
|
2019-03-02 00:08:59 +07:00
|
|
|
rq->rcustate = get_state_synchronize_rcu(); /* acts as smp_mb() */
|
2016-10-28 19:58:46 +07:00
|
|
|
|
2019-03-02 00:08:59 +07:00
|
|
|
dma_fence_init(&rq->fence, &i915_fence_ops, &rq->lock,
|
|
|
|
tl->fence_context, seqno);
|
2016-07-20 15:21:11 +07:00
|
|
|
|
2016-11-25 20:17:17 +07:00
|
|
|
/* We bump the ref for the fence chain */
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
i915_sw_fence_reinit(&i915_request_get(rq)->submit);
|
|
|
|
i915_sw_fence_reinit(&i915_request_get(rq)->semaphore);
|
2016-09-09 20:11:54 +07:00
|
|
|
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
i915_sched_node_reinit(&rq->sched);
|
2016-11-15 03:41:02 +07:00
|
|
|
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
/* No zalloc, everything must be cleared after use */
|
2018-02-21 16:56:36 +07:00
|
|
|
rq->batch = NULL;
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
GEM_BUG_ON(rq->file_priv);
|
|
|
|
GEM_BUG_ON(rq->capture_list);
|
|
|
|
GEM_BUG_ON(!list_empty(&rq->execute_cb));
|
2019-04-25 03:07:16 +07:00
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/*
|
|
|
|
* Reserve space in the ring buffer for all the commands required to
|
|
|
|
* eventually emit this request. This is to guarantee that the
|
2018-02-21 16:56:36 +07:00
|
|
|
* i915_request_add() call can't fail. Note that the reserve may need
|
2016-07-20 15:21:08 +07:00
|
|
|
* to be redone if the request is not actually submitted straight
|
|
|
|
* away, e.g. because a GPU scheduler has deferred it.
|
2018-12-29 00:16:36 +07:00
|
|
|
*
|
|
|
|
* Note that due to how we add reserved_space to intel_ring_begin()
|
|
|
|
* we need to double our request to ensure that if we need to wrap
|
|
|
|
* around inside i915_request_add() there is sufficient space at
|
|
|
|
* the beginning of the ring as well.
|
2016-07-20 15:21:08 +07:00
|
|
|
*/
|
2019-04-25 03:07:16 +07:00
|
|
|
rq->reserved_space =
|
|
|
|
2 * rq->engine->emit_fini_breadcrumb_dw * sizeof(u32);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2017-11-20 17:20:01 +07:00
|
|
|
/*
|
|
|
|
* Record the position of the start of the request so that
|
2016-08-15 16:48:40 +07:00
|
|
|
* should we detect the updated seqno part-way through the
|
|
|
|
* GPU processing the request, we never over-estimate the
|
|
|
|
* position of the head.
|
|
|
|
*/
|
2018-02-21 16:56:36 +07:00
|
|
|
rq->head = rq->ring->emit;
|
2016-08-15 16:48:40 +07:00
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
ret = rq->engine->request_alloc(rq);
|
2017-11-23 22:26:30 +07:00
|
|
|
if (ret)
|
|
|
|
goto err_unwind;
|
2017-11-20 17:20:01 +07:00
|
|
|
|
2018-06-11 18:08:44 +07:00
|
|
|
rq->infix = rq->ring->emit; /* end of header; start of user payload */
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
intel_context_mark_active(ce);
|
2018-02-21 16:56:36 +07:00
|
|
|
return rq;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2017-11-23 22:26:30 +07:00
|
|
|
err_unwind:
|
2018-05-18 04:26:32 +07:00
|
|
|
ce->ring->emit = rq->head;
|
2017-11-23 22:26:30 +07:00
|
|
|
|
2016-11-25 20:17:16 +07:00
|
|
|
/* Make sure we didn't add ourselves to external state before freeing */
|
2018-04-19 01:40:51 +07:00
|
|
|
GEM_BUG_ON(!list_empty(&rq->sched.signalers_list));
|
|
|
|
GEM_BUG_ON(!list_empty(&rq->sched.waiters_list));
|
2016-11-25 20:17:16 +07:00
|
|
|
|
2019-03-02 00:08:59 +07:00
|
|
|
err_free:
|
2019-02-28 17:20:33 +07:00
|
|
|
kmem_cache_free(global.slab_requests, rq);
|
2016-10-28 19:58:56 +07:00
|
|
|
err_unreserve:
|
2018-05-18 04:26:32 +07:00
|
|
|
intel_context_unpin(ce);
|
2016-08-03 04:50:26 +07:00
|
|
|
return ERR_PTR(ret);
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
struct i915_request *
|
|
|
|
i915_request_create(struct intel_context *ce)
|
|
|
|
{
|
|
|
|
struct i915_request *rq;
|
2019-08-16 03:57:09 +07:00
|
|
|
struct intel_timeline *tl;
|
2019-04-25 03:07:16 +07:00
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
tl = intel_context_timeline_lock(ce);
|
|
|
|
if (IS_ERR(tl))
|
|
|
|
return ERR_CAST(tl);
|
2019-04-25 03:07:16 +07:00
|
|
|
|
|
|
|
/* Move our oldest request to the slab-cache (if not in use!) */
|
2019-08-16 03:57:09 +07:00
|
|
|
rq = list_first_entry(&tl->requests, typeof(*rq), link);
|
|
|
|
if (!list_is_last(&rq->link, &tl->requests))
|
2019-04-25 03:07:16 +07:00
|
|
|
i915_request_retire(rq);
|
|
|
|
|
|
|
|
intel_context_enter(ce);
|
|
|
|
rq = __i915_request_create(ce, GFP_KERNEL);
|
|
|
|
intel_context_exit(ce); /* active reference transferred to request */
|
|
|
|
if (IS_ERR(rq))
|
|
|
|
goto err_unlock;
|
|
|
|
|
|
|
|
/* Check that we do not interrupt ourselves with a new request */
|
2019-08-16 03:57:09 +07:00
|
|
|
rq->cookie = lockdep_pin_lock(&tl->mutex);
|
2019-04-25 03:07:16 +07:00
|
|
|
|
|
|
|
return rq;
|
|
|
|
|
|
|
|
err_unlock:
|
2019-08-16 03:57:09 +07:00
|
|
|
intel_context_timeline_unlock(tl);
|
2019-04-25 03:07:16 +07:00
|
|
|
return rq;
|
|
|
|
}
|
|
|
|
|
2019-05-01 18:45:36 +07:00
|
|
|
static int
|
|
|
|
i915_request_await_start(struct i915_request *rq, struct i915_request *signal)
|
|
|
|
{
|
2019-09-19 18:19:11 +07:00
|
|
|
struct dma_fence *fence;
|
|
|
|
int err;
|
2019-05-01 18:45:36 +07:00
|
|
|
|
2019-09-19 18:19:11 +07:00
|
|
|
GEM_BUG_ON(i915_request_timeline(rq) ==
|
|
|
|
rcu_access_pointer(signal->timeline));
|
|
|
|
|
2019-12-16 23:53:17 +07:00
|
|
|
fence = NULL;
|
2019-09-19 18:19:11 +07:00
|
|
|
rcu_read_lock();
|
2019-12-16 23:53:17 +07:00
|
|
|
spin_lock_irq(&signal->lock);
|
|
|
|
if (!i915_request_started(signal) &&
|
|
|
|
!list_is_first(&signal->link,
|
|
|
|
&rcu_dereference(signal->timeline)->requests)) {
|
|
|
|
struct i915_request *prev = list_prev_entry(signal, link);
|
2019-05-01 18:45:36 +07:00
|
|
|
|
2019-12-16 23:53:17 +07:00
|
|
|
/*
|
|
|
|
* Peek at the request before us in the timeline. That
|
|
|
|
* request will only be valid before it is retired, so
|
|
|
|
* after acquiring a reference to it, confirm that it is
|
|
|
|
* still part of the signaler's timeline.
|
|
|
|
*/
|
|
|
|
if (i915_request_get_rcu(prev)) {
|
|
|
|
if (list_next_entry(prev, link) == signal)
|
|
|
|
fence = &prev->fence;
|
|
|
|
else
|
|
|
|
i915_request_put(prev);
|
2019-09-19 18:19:11 +07:00
|
|
|
}
|
|
|
|
}
|
2019-12-16 23:53:17 +07:00
|
|
|
spin_unlock_irq(&signal->lock);
|
|
|
|
rcu_read_unlock();
|
|
|
|
if (!fence)
|
|
|
|
return 0;
|
2019-09-19 18:19:11 +07:00
|
|
|
|
|
|
|
err = 0;
|
|
|
|
if (intel_timeline_sync_is_later(i915_request_timeline(rq), fence))
|
|
|
|
err = i915_sw_fence_await_dma_fence(&rq->submit,
|
|
|
|
fence, 0,
|
|
|
|
I915_FENCE_GFP);
|
|
|
|
dma_fence_put(fence);
|
|
|
|
|
|
|
|
return err;
|
2019-05-01 18:45:36 +07:00
|
|
|
}
|
|
|
|
|
drm/i915: Disable semaphore busywaits on saturated systems
Asking the GPU to busywait on a memory address, perhaps not unexpectedly
in hindsight for a shared system, leads to bus contention that affects
CPU programs trying to concurrently access memory. This can manifest as
a drop in transcode throughput on highly over-saturated workloads.
The only clue offered by perf, is that the bus-cycles (perf stat -e
bus-cycles) jumped by 50% when enabling semaphores. This corresponds
with extra CPU active cycles being attributed to intel_idle's mwait.
This patch introduces a heuristic to try and detect when more than one
client is submitting to the GPU pushing it into an oversaturated state.
As we already keep track of when the semaphores are signaled, we can
inspect their state on submitting the busywait batch and if we planned
to use a semaphore but were too late, conclude that the GPU is
overloaded and not try to use semaphores in future requests. In
practice, this means we optimistically try to use semaphores for the
first frame of a transcode job split over multiple engines, and fail if
there are multiple clients active and continue not to use semaphores for
the subsequent frames in the sequence. Periodically, we try to
optimistically switch semaphores back on whenever the client waits to
catch up with the transcode results.
With 1 client, on Broxton J3455, with the relative fps normalized by %cpu:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| * |
| *+ |
| **+ |
| **+ x |
| x * +**+ x |
| x x * * +***x xx |
| x x * * *+***x *x |
| x x* + * * *****x *x x |
| + x xx+x* + *** * ********* x * |
| + x xx+x* * *** +** ********* xx * |
| * + ++++* + x*x****+*+* ***+*************+x* * |
|*+ +** *+ + +* + *++****** *xxx**********x***+*****************+*++ *|
| |__________A_____M_____| |
| |_______________A____M_________| |
| |____________A___M________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.60475 3.50941 3.31123 3.2143953 0.21117399
+ 120 2.3826 3.57077 3.25101 3.1414161 0.28146407
Difference at 95.0% confidence
-0.0729792 +/- 0.0629585
-2.27039% +/- 1.95864%
(Student's t, pooled s = 0.248814)
* 120 2.35536 3.66713 3.2849 3.2059917 0.24618565
No difference proven at 95.0% confidence
With 10 clients over-saturating the pipeline:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| ++ ** |
| ++ ** |
| ++ ** |
| ++ ** |
| ++ xx *** |
| ++ xx *** |
| ++ xxx*** |
| ++ xxx*** |
| +++ xxx*** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| ++++ xx**** |
| +++++ xx**** |
| +++++ x x****** |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xx******** |
| ++++++ xxxx******** |
| ++++++ xxxx******** |
| ++++++++ xxxxx********* |
|+ + + + ++++++++ xxx*xx**********x* *|
| |__A__| |
| |__AM__| |
| |__A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.47855 2.8972 2.72376 2.7193402 0.074604933
+ 120 1.17367 1.77459 1.71977 1.6966782 0.085850697
Difference at 95.0% confidence
-1.02266 +/- 0.0203502
-37.607% +/- 0.748352%
(Student's t, pooled s = 0.0804246)
* 120 2.57868 3.00821 2.80142 2.7923878 0.058646477
Difference at 95.0% confidence
0.0730476 +/- 0.0169791
2.68622% +/- 0.624383%
(Student's t, pooled s = 0.0671018)
Indicating that we've recovered the regression from enabling semaphores
on this saturated setup, with a hint towards an overall improvement.
Very similar, but of smaller magnitude, results are observed on both
Skylake(gt2) and Kabylake(gt4). This may be due to the reduced impact of
bus-cycles, where we see a 50% hit on Broxton, it is only 10% on the big
core, in this particular test.
One observation to make here is that for a greedy client trying to
maximise its own throughput, using semaphores is the right choice. It is
only the holistic system-wide view that semaphores of one client
impacts another and reduces the overall throughput where we would choose
to disable semaphores.
The most noticeable negactive impact this has is on the no-op
microbenchmarks, which are also very notable for having no cpu bus load.
In particular, this increases the runtime and energy consumption of
gem_exec_whisper.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190504070707.30902-1-chris@chris-wilson.co.uk
2019-05-04 14:07:07 +07:00
|
|
|
static intel_engine_mask_t
|
|
|
|
already_busywaiting(struct i915_request *rq)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Polling a semaphore causes bus traffic, delaying other users of
|
|
|
|
* both the GPU and CPU. We want to limit the impact on others,
|
|
|
|
* while taking advantage of early submission to reduce GPU
|
|
|
|
* latency. Therefore we restrict ourselves to not using more
|
|
|
|
* than one semaphore from each source, and not using a semaphore
|
|
|
|
* if we have detected the engine is saturated (i.e. would not be
|
|
|
|
* submitted early and cause bus traffic reading an already passed
|
|
|
|
* semaphore).
|
|
|
|
*
|
|
|
|
* See the are-we-too-late? check in __i915_request_submit().
|
|
|
|
*/
|
2019-06-18 14:41:35 +07:00
|
|
|
return rq->sched.semaphores | rq->engine->saturated;
|
drm/i915: Disable semaphore busywaits on saturated systems
Asking the GPU to busywait on a memory address, perhaps not unexpectedly
in hindsight for a shared system, leads to bus contention that affects
CPU programs trying to concurrently access memory. This can manifest as
a drop in transcode throughput on highly over-saturated workloads.
The only clue offered by perf, is that the bus-cycles (perf stat -e
bus-cycles) jumped by 50% when enabling semaphores. This corresponds
with extra CPU active cycles being attributed to intel_idle's mwait.
This patch introduces a heuristic to try and detect when more than one
client is submitting to the GPU pushing it into an oversaturated state.
As we already keep track of when the semaphores are signaled, we can
inspect their state on submitting the busywait batch and if we planned
to use a semaphore but were too late, conclude that the GPU is
overloaded and not try to use semaphores in future requests. In
practice, this means we optimistically try to use semaphores for the
first frame of a transcode job split over multiple engines, and fail if
there are multiple clients active and continue not to use semaphores for
the subsequent frames in the sequence. Periodically, we try to
optimistically switch semaphores back on whenever the client waits to
catch up with the transcode results.
With 1 client, on Broxton J3455, with the relative fps normalized by %cpu:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| * |
| *+ |
| **+ |
| **+ x |
| x * +**+ x |
| x x * * +***x xx |
| x x * * *+***x *x |
| x x* + * * *****x *x x |
| + x xx+x* + *** * ********* x * |
| + x xx+x* * *** +** ********* xx * |
| * + ++++* + x*x****+*+* ***+*************+x* * |
|*+ +** *+ + +* + *++****** *xxx**********x***+*****************+*++ *|
| |__________A_____M_____| |
| |_______________A____M_________| |
| |____________A___M________| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.60475 3.50941 3.31123 3.2143953 0.21117399
+ 120 2.3826 3.57077 3.25101 3.1414161 0.28146407
Difference at 95.0% confidence
-0.0729792 +/- 0.0629585
-2.27039% +/- 1.95864%
(Student's t, pooled s = 0.248814)
* 120 2.35536 3.66713 3.2849 3.2059917 0.24618565
No difference proven at 95.0% confidence
With 10 clients over-saturating the pipeline:
x no semaphores
+ drm-tip
* patched
+------------------------------------------------------------------------+
| ++ ** |
| ++ ** |
| ++ ** |
| ++ ** |
| ++ xx *** |
| ++ xx *** |
| ++ xxx*** |
| ++ xxx*** |
| +++ xxx*** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| +++ xx**** |
| ++++ xx**** |
| +++++ xx**** |
| +++++ x x****** |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xxx******* |
| ++++++ xx******** |
| ++++++ xxxx******** |
| ++++++ xxxx******** |
| ++++++++ xxxxx********* |
|+ + + + ++++++++ xxx*xx**********x* *|
| |__A__| |
| |__AM__| |
| |__A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 2.47855 2.8972 2.72376 2.7193402 0.074604933
+ 120 1.17367 1.77459 1.71977 1.6966782 0.085850697
Difference at 95.0% confidence
-1.02266 +/- 0.0203502
-37.607% +/- 0.748352%
(Student's t, pooled s = 0.0804246)
* 120 2.57868 3.00821 2.80142 2.7923878 0.058646477
Difference at 95.0% confidence
0.0730476 +/- 0.0169791
2.68622% +/- 0.624383%
(Student's t, pooled s = 0.0671018)
Indicating that we've recovered the regression from enabling semaphores
on this saturated setup, with a hint towards an overall improvement.
Very similar, but of smaller magnitude, results are observed on both
Skylake(gt2) and Kabylake(gt4). This may be due to the reduced impact of
bus-cycles, where we see a 50% hit on Broxton, it is only 10% on the big
core, in this particular test.
One observation to make here is that for a greedy client trying to
maximise its own throughput, using semaphores is the right choice. It is
only the holistic system-wide view that semaphores of one client
impacts another and reduces the overall throughput where we would choose
to disable semaphores.
The most noticeable negactive impact this has is on the no-op
microbenchmarks, which are also very notable for having no cpu bus load.
In particular, this increases the runtime and energy consumption of
gem_exec_whisper.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190504070707.30902-1-chris@chris-wilson.co.uk
2019-05-04 14:07:07 +07:00
|
|
|
}
|
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
static int
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
__emit_semaphore_wait(struct i915_request *to,
|
|
|
|
struct i915_request *from,
|
|
|
|
u32 seqno)
|
2019-03-02 00:09:00 +07:00
|
|
|
{
|
2019-09-17 19:30:55 +07:00
|
|
|
const int has_token = INTEL_GEN(to->i915) >= 12;
|
2019-03-02 00:09:00 +07:00
|
|
|
u32 hwsp_offset;
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
int len, err;
|
2019-03-02 00:09:00 +07:00
|
|
|
u32 *cs;
|
|
|
|
|
|
|
|
GEM_BUG_ON(INTEL_GEN(to->i915) < 8);
|
|
|
|
|
2019-05-03 21:02:39 +07:00
|
|
|
/* We need to pin the signaler's HWSP until we are finished reading. */
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
err = intel_timeline_read_hwsp(from, to, &hwsp_offset);
|
|
|
|
if (err)
|
|
|
|
return err;
|
2019-03-02 00:09:00 +07:00
|
|
|
|
2019-09-17 19:30:55 +07:00
|
|
|
len = 4;
|
|
|
|
if (has_token)
|
|
|
|
len += 2;
|
|
|
|
|
|
|
|
cs = intel_ring_begin(to, len);
|
2019-03-02 00:09:00 +07:00
|
|
|
if (IS_ERR(cs))
|
|
|
|
return PTR_ERR(cs);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Using greater-than-or-equal here means we have to worry
|
|
|
|
* about seqno wraparound. To side step that issue, we swap
|
|
|
|
* the timeline HWSP upon wrapping, so that everyone listening
|
|
|
|
* for the old (pre-wrap) values do not see the much smaller
|
|
|
|
* (post-wrap) values than they were expecting (and so wait
|
|
|
|
* forever).
|
|
|
|
*/
|
2019-09-17 19:30:55 +07:00
|
|
|
*cs++ = (MI_SEMAPHORE_WAIT |
|
|
|
|
MI_SEMAPHORE_GLOBAL_GTT |
|
|
|
|
MI_SEMAPHORE_POLL |
|
|
|
|
MI_SEMAPHORE_SAD_GTE_SDD) +
|
|
|
|
has_token;
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
*cs++ = seqno;
|
2019-03-02 00:09:00 +07:00
|
|
|
*cs++ = hwsp_offset;
|
|
|
|
*cs++ = 0;
|
2019-09-17 19:30:55 +07:00
|
|
|
if (has_token) {
|
|
|
|
*cs++ = 0;
|
|
|
|
*cs++ = MI_NOOP;
|
|
|
|
}
|
2019-03-02 00:09:00 +07:00
|
|
|
|
|
|
|
intel_ring_advance(to, cs);
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
|
|
|
emit_semaphore_wait(struct i915_request *to,
|
|
|
|
struct i915_request *from,
|
|
|
|
gfp_t gfp)
|
|
|
|
{
|
|
|
|
/* Just emit the first semaphore we see as request space is limited. */
|
|
|
|
if (already_busywaiting(to) & from->engine->mask)
|
|
|
|
goto await_fence;
|
|
|
|
|
|
|
|
if (i915_request_await_start(to, from) < 0)
|
|
|
|
goto await_fence;
|
|
|
|
|
|
|
|
/* Only submit our spinner after the signaler is running! */
|
|
|
|
if (__await_execution(to, from, NULL, gfp))
|
|
|
|
goto await_fence;
|
|
|
|
|
|
|
|
if (__emit_semaphore_wait(to, from, from->fence.seqno))
|
|
|
|
goto await_fence;
|
|
|
|
|
2019-04-01 23:26:41 +07:00
|
|
|
to->sched.semaphores |= from->engine->mask;
|
|
|
|
to->sched.flags |= I915_SCHED_HAS_SEMAPHORE_CHAIN;
|
2019-03-02 00:09:00 +07:00
|
|
|
return 0;
|
2019-09-19 18:19:11 +07:00
|
|
|
|
|
|
|
await_fence:
|
|
|
|
return i915_sw_fence_await_dma_fence(&to->submit,
|
|
|
|
&from->fence, 0,
|
|
|
|
I915_FENCE_GFP);
|
2019-03-02 00:09:00 +07:00
|
|
|
}
|
|
|
|
|
2016-09-09 20:11:56 +07:00
|
|
|
static int
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_request_await_request(struct i915_request *to, struct i915_request *from)
|
2016-09-09 20:11:56 +07:00
|
|
|
{
|
2016-10-28 19:58:53 +07:00
|
|
|
int ret;
|
2016-09-09 20:11:56 +07:00
|
|
|
|
|
|
|
GEM_BUG_ON(to == from);
|
2017-05-03 16:39:20 +07:00
|
|
|
GEM_BUG_ON(to->timeline == from->timeline);
|
2016-09-09 20:11:56 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
if (i915_request_completed(from))
|
2017-04-22 15:15:37 +07:00
|
|
|
return 0;
|
|
|
|
|
2016-11-15 03:41:02 +07:00
|
|
|
if (to->engine->schedule) {
|
2019-02-28 17:20:33 +07:00
|
|
|
ret = i915_sched_node_add_dependency(&to->sched, &from->sched);
|
2016-11-15 03:41:02 +07:00
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2019-12-20 17:12:30 +07:00
|
|
|
if (to->engine == from->engine)
|
2016-10-28 19:58:46 +07:00
|
|
|
ret = i915_sw_fence_await_sw_fence_gfp(&to->submit,
|
|
|
|
&from->submit,
|
2017-12-13 01:06:51 +07:00
|
|
|
I915_FENCE_GFP);
|
2019-12-20 17:12:30 +07:00
|
|
|
else if (intel_context_use_semaphores(to->context))
|
2019-03-02 00:09:00 +07:00
|
|
|
ret = emit_semaphore_wait(to, from, I915_FENCE_GFP);
|
2019-12-20 17:12:30 +07:00
|
|
|
else
|
2018-12-28 21:07:35 +07:00
|
|
|
ret = i915_sw_fence_await_dma_fence(&to->submit,
|
|
|
|
&from->fence, 0,
|
|
|
|
I915_FENCE_GFP);
|
drm/i915: Truly bump ready tasks ahead of busywaits
In commit b7404c7ecb38 ("drm/i915: Bump ready tasks ahead of
busywaits"), I tried cutting a corner in order to not install a signal
for each of our dependencies, and only listened to requests on which we
were intending to busywait. The compromise that was made was that
instead of then being able to promote the request with a full
NOSEMAPHORE like its non-busywaiting brethren, as we had not ensured we
had cleared the semaphore chain, we settled for only using the NEWCLIENT
boost. With an over saturated system with multiple NEWCLIENTS in flight
at any time, this was found to be an inadequate promotion and left us
with a much poorer scheduling order than prior to using semaphores.
The outcome of this patch, is that all requests have NOSEMAPHORE
priority when they have no dependencies and are ready to run and not
busywait, restoring the pre-semaphore ordering on saturated systems.
We can demonstrate the effect of poor scheduling order by oversaturating
the system using gem_wsim on a system with multiple vcs engines
(i.e running the same workloads across more clients than required for
peak throughput, e.g. media_load_balance_17i7.wsim -c4 -b context):
x v5.1 (normalized)
+ tip
* fix
+------------------------------------------------------------------------+
| x |
| x |
| x |
| x |
| %x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %#x |
| %#x |
| %#x |
| %#x |
| %#x |
| + %#xx |
| + %#xx |
| + %%#xx |
| + %%#xx |
| + %%#xx |
| + %%#xx |
| + %%##x |
| +++ %%##x |
| +++ %%##x |
| +++ %%##x |
| ++++ %%##x |
| ++++ %%##x |
| ++++ %%##xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ + %#O#xx |
| ++++ + %#O#xx |
| ++++++ + %#O#xx |
| ++++++++++ %OOOxxx|
| ++++++++++ + %#OOO#xx|
| + ++++++++++++ ++ +++++ + ++ @@OOOO#xx|
| |A_| |
||__________M_______A____________________| |
| |A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 0.99456 1.00628 0.999985 1.0001545 0.0024387139
+ 120 0.873021 1.00037 0.884134 0.90148752 0.039190862
Difference at 99.5% confidence
-0.098667 +/- 0.0110762
-9.86517% +/- 1.10745%
(Student's t, pooled s = 0.0277657)
% 120 0.990207 1.00165 0.9970265 0.99699748 0.0021024
Difference at 99.5% confidence
-0.003157 +/- 0.000908245
-0.315651% +/- 0.0908105%
(Student's t, pooled s = 0.00227678)
Fixes: b7404c7ecb38 ("drm/i915: Bump ready tasks ahead of busywaits")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190515130052.4475-2-chris@chris-wilson.co.uk
2019-05-15 20:00:49 +07:00
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
if (to->sched.flags & I915_SCHED_HAS_SEMAPHORE_CHAIN) {
|
|
|
|
ret = i915_sw_fence_await_dma_fence(&to->semaphore,
|
|
|
|
&from->fence, 0,
|
|
|
|
I915_FENCE_GFP);
|
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
}
|
2016-09-09 20:11:56 +07:00
|
|
|
|
drm/i915: Truly bump ready tasks ahead of busywaits
In commit b7404c7ecb38 ("drm/i915: Bump ready tasks ahead of
busywaits"), I tried cutting a corner in order to not install a signal
for each of our dependencies, and only listened to requests on which we
were intending to busywait. The compromise that was made was that
instead of then being able to promote the request with a full
NOSEMAPHORE like its non-busywaiting brethren, as we had not ensured we
had cleared the semaphore chain, we settled for only using the NEWCLIENT
boost. With an over saturated system with multiple NEWCLIENTS in flight
at any time, this was found to be an inadequate promotion and left us
with a much poorer scheduling order than prior to using semaphores.
The outcome of this patch, is that all requests have NOSEMAPHORE
priority when they have no dependencies and are ready to run and not
busywait, restoring the pre-semaphore ordering on saturated systems.
We can demonstrate the effect of poor scheduling order by oversaturating
the system using gem_wsim on a system with multiple vcs engines
(i.e running the same workloads across more clients than required for
peak throughput, e.g. media_load_balance_17i7.wsim -c4 -b context):
x v5.1 (normalized)
+ tip
* fix
+------------------------------------------------------------------------+
| x |
| x |
| x |
| x |
| %x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %%x |
| %#x |
| %#x |
| %#x |
| %#x |
| %#x |
| + %#xx |
| + %#xx |
| + %%#xx |
| + %%#xx |
| + %%#xx |
| + %%#xx |
| + %%##x |
| +++ %%##x |
| +++ %%##x |
| +++ %%##x |
| ++++ %%##x |
| ++++ %%##x |
| ++++ %%##xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ %###xx |
| ++++ + %#O#xx |
| ++++ + %#O#xx |
| ++++++ + %#O#xx |
| ++++++++++ %OOOxxx|
| ++++++++++ + %#OOO#xx|
| + ++++++++++++ ++ +++++ + ++ @@OOOO#xx|
| |A_| |
||__________M_______A____________________| |
| |A_| |
+------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 120 0.99456 1.00628 0.999985 1.0001545 0.0024387139
+ 120 0.873021 1.00037 0.884134 0.90148752 0.039190862
Difference at 99.5% confidence
-0.098667 +/- 0.0110762
-9.86517% +/- 1.10745%
(Student's t, pooled s = 0.0277657)
% 120 0.990207 1.00165 0.9970265 0.99699748 0.0021024
Difference at 99.5% confidence
-0.003157 +/- 0.000908245
-0.315651% +/- 0.0908105%
(Student's t, pooled s = 0.00227678)
Fixes: b7404c7ecb38 ("drm/i915: Bump ready tasks ahead of busywaits")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Cc: Dmitry Ermilov <dmitry.ermilov@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190515130052.4475-2-chris@chris-wilson.co.uk
2019-05-15 20:00:49 +07:00
|
|
|
return 0;
|
2016-09-09 20:11:56 +07:00
|
|
|
}
|
|
|
|
|
2016-10-28 19:58:24 +07:00
|
|
|
int
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_request_await_dma_fence(struct i915_request *rq, struct dma_fence *fence)
|
2016-10-28 19:58:24 +07:00
|
|
|
{
|
2017-05-03 16:39:19 +07:00
|
|
|
struct dma_fence **child = &fence;
|
|
|
|
unsigned int nchild = 1;
|
2016-10-28 19:58:24 +07:00
|
|
|
int ret;
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* Note that if the fence-array was created in signal-on-any mode,
|
2016-10-28 19:58:24 +07:00
|
|
|
* we should *not* decompose it into its individual fences. However,
|
|
|
|
* we don't currently store which mode the fence-array is operating
|
|
|
|
* in. Fortunately, the only user of signal-on-any is private to
|
|
|
|
* amdgpu and we should not see any incoming fence-array from
|
|
|
|
* sync-file being in signal-on-any mode.
|
|
|
|
*/
|
2017-05-03 16:39:19 +07:00
|
|
|
if (dma_fence_is_array(fence)) {
|
|
|
|
struct dma_fence_array *array = to_dma_fence_array(fence);
|
|
|
|
|
|
|
|
child = array->fences;
|
|
|
|
nchild = array->num_fences;
|
|
|
|
GEM_BUG_ON(!nchild);
|
|
|
|
}
|
2016-10-28 19:58:24 +07:00
|
|
|
|
2017-05-03 16:39:19 +07:00
|
|
|
do {
|
|
|
|
fence = *child++;
|
2019-12-06 23:04:27 +07:00
|
|
|
if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
|
|
|
|
i915_sw_fence_set_error_once(&rq->submit, fence->error);
|
2017-05-03 16:39:19 +07:00
|
|
|
continue;
|
2019-12-06 23:04:27 +07:00
|
|
|
}
|
2016-10-28 19:58:24 +07:00
|
|
|
|
2017-05-03 16:39:20 +07:00
|
|
|
/*
|
|
|
|
* Requests on the same timeline are explicitly ordered, along
|
2018-02-21 16:56:36 +07:00
|
|
|
* with their dependencies, by i915_request_add() which ensures
|
2017-05-03 16:39:20 +07:00
|
|
|
* that requests are submitted in-order through each ring.
|
|
|
|
*/
|
2018-02-21 16:56:36 +07:00
|
|
|
if (fence->context == rq->fence.context)
|
2017-05-03 16:39:20 +07:00
|
|
|
continue;
|
|
|
|
|
2017-05-03 16:39:21 +07:00
|
|
|
/* Squash repeated waits to the same timelines */
|
2019-08-20 01:44:03 +07:00
|
|
|
if (fence->context &&
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
intel_timeline_sync_is_later(i915_request_timeline(rq),
|
|
|
|
fence))
|
2017-05-03 16:39:21 +07:00
|
|
|
continue;
|
|
|
|
|
2017-05-03 16:39:19 +07:00
|
|
|
if (dma_fence_is_i915(fence))
|
2018-02-21 16:56:36 +07:00
|
|
|
ret = i915_request_await_request(rq, to_request(fence));
|
2016-10-28 19:58:24 +07:00
|
|
|
else
|
2018-02-21 16:56:36 +07:00
|
|
|
ret = i915_sw_fence_await_dma_fence(&rq->submit, fence,
|
2019-08-26 14:21:27 +07:00
|
|
|
fence->context ? I915_FENCE_TIMEOUT : 0,
|
2017-12-13 01:06:51 +07:00
|
|
|
I915_FENCE_GFP);
|
2016-10-28 19:58:24 +07:00
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
2017-05-03 16:39:21 +07:00
|
|
|
|
|
|
|
/* Record the latest fence used against each timeline */
|
2019-08-20 01:44:03 +07:00
|
|
|
if (fence->context)
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
intel_timeline_sync_set(i915_request_timeline(rq),
|
|
|
|
fence);
|
2017-05-03 16:39:19 +07:00
|
|
|
} while (--nchild);
|
2016-10-28 19:58:24 +07:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
static bool intel_timeline_sync_has_start(struct intel_timeline *tl,
|
|
|
|
struct dma_fence *fence)
|
|
|
|
{
|
|
|
|
return __intel_timeline_sync_is_later(tl,
|
|
|
|
fence->context,
|
|
|
|
fence->seqno - 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int intel_timeline_sync_set_start(struct intel_timeline *tl,
|
|
|
|
const struct dma_fence *fence)
|
|
|
|
{
|
|
|
|
return __intel_timeline_sync_set(tl, fence->context, fence->seqno - 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
|
|
|
__i915_request_await_execution(struct i915_request *to,
|
|
|
|
struct i915_request *from,
|
|
|
|
void (*hook)(struct i915_request *rq,
|
|
|
|
struct dma_fence *signal))
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
|
|
|
|
/* Submit both requests at the same time */
|
|
|
|
err = __await_execution(to, from, hook, I915_FENCE_GFP);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
/* Squash repeated depenendices to the same timelines */
|
|
|
|
if (intel_timeline_sync_has_start(i915_request_timeline(to),
|
|
|
|
&from->fence))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Ensure both start together [after all semaphores in signal] */
|
|
|
|
if (intel_engine_has_semaphores(to->engine))
|
|
|
|
err = __emit_semaphore_wait(to, from, from->fence.seqno - 1);
|
|
|
|
else
|
|
|
|
err = i915_request_await_start(to, from);
|
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
/* Couple the dependency tree for PI on this exposed to->fence */
|
|
|
|
if (to->engine->schedule) {
|
|
|
|
err = i915_sched_node_add_dependency(&to->sched, &from->sched);
|
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
return intel_timeline_sync_set_start(i915_request_timeline(to),
|
|
|
|
&from->fence);
|
|
|
|
}
|
|
|
|
|
2019-05-22 04:11:32 +07:00
|
|
|
int
|
|
|
|
i915_request_await_execution(struct i915_request *rq,
|
|
|
|
struct dma_fence *fence,
|
|
|
|
void (*hook)(struct i915_request *rq,
|
|
|
|
struct dma_fence *signal))
|
|
|
|
{
|
|
|
|
struct dma_fence **child = &fence;
|
|
|
|
unsigned int nchild = 1;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (dma_fence_is_array(fence)) {
|
|
|
|
struct dma_fence_array *array = to_dma_fence_array(fence);
|
|
|
|
|
|
|
|
/* XXX Error for signal-on-any fence arrays */
|
|
|
|
|
|
|
|
child = array->fences;
|
|
|
|
nchild = array->num_fences;
|
|
|
|
GEM_BUG_ON(!nchild);
|
|
|
|
}
|
|
|
|
|
|
|
|
do {
|
|
|
|
fence = *child++;
|
2019-12-06 23:04:27 +07:00
|
|
|
if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
|
|
|
|
i915_sw_fence_set_error_once(&rq->submit, fence->error);
|
2019-05-22 04:11:32 +07:00
|
|
|
continue;
|
2019-12-06 23:04:27 +07:00
|
|
|
}
|
2019-05-22 04:11:32 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We don't squash repeated fence dependencies here as we
|
|
|
|
* want to run our callback in all cases.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (dma_fence_is_i915(fence))
|
|
|
|
ret = __i915_request_await_execution(rq,
|
|
|
|
to_request(fence),
|
drm/i915: Copy across scheduler behaviour flags across submit fences
We want the bonded request to have the same scheduler properties as its
master so that it is placed at the same depth in the queue. For example,
consider we have requests A, B and B', where B & B' are a bonded pair to
run in parallel on two engines.
A -> B
\- B'
B will run after A and so may be scheduled on an idle engine and wait on
A using a semaphore. B' sees B being executed and so enters the queue on
the same engine as A. As B' did not inherit the semaphore-chain from B,
it may have higher precedence than A and so preempts execution. However,
B' then sits on a semaphore waiting for B, who is waiting for A, who is
blocked by B.
Ergo B' needs to inherit the scheduler properties from B (i.e. the
semaphore chain) so that it is scheduled with the same priority as B and
will not be executed ahead of Bs dependencies.
Furthermore, to prevent the priorities changing via the expose fence on
B', we need to couple in the dependencies for PI. This requires us to
relax our sanity-checks that dependencies are strictly in order.
v2: Synchronise (B, B') execution on all platforms, regardless of using
a scheduler, any no-op syncs should be elided.
Fixes: ee1136908e9b ("drm/i915/execlists: Virtual engine bonding")
Closes: https://gitlab.freedesktop.org/drm/intel/issues/464
Testcase: igt/gem_exec_balancer/bonded-chain
Testcase: igt/gem_exec_balancer/bonded-semaphore
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191210151332.3902215-1-chris@chris-wilson.co.uk
2019-12-10 22:13:32 +07:00
|
|
|
hook);
|
2019-05-22 04:11:32 +07:00
|
|
|
else
|
|
|
|
ret = i915_sw_fence_await_dma_fence(&rq->submit, fence,
|
|
|
|
I915_FENCE_TIMEOUT,
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
} while (--nchild);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-09-09 20:11:56 +07:00
|
|
|
/**
|
2018-02-21 16:56:36 +07:00
|
|
|
* i915_request_await_object - set this request to (async) wait upon a bo
|
2016-09-09 20:11:56 +07:00
|
|
|
* @to: request we are wishing to use
|
|
|
|
* @obj: object which may be in use on another ring.
|
2018-02-08 18:14:53 +07:00
|
|
|
* @write: whether the wait is on behalf of a writer
|
2016-09-09 20:11:56 +07:00
|
|
|
*
|
|
|
|
* This code is meant to abstract object synchronization with the GPU.
|
|
|
|
* Conceptually we serialise writes between engines inside the GPU.
|
|
|
|
* We only allow one engine to write into a buffer at any time, but
|
|
|
|
* multiple readers. To ensure each has a coherent view of memory, we must:
|
|
|
|
*
|
|
|
|
* - If there is an outstanding write request to the object, the new
|
|
|
|
* request must wait for it to complete (either CPU or in hw, requests
|
|
|
|
* on the same ring will be naturally ordered).
|
|
|
|
*
|
|
|
|
* - If we are a write request (pending_write_domain is set), the new
|
|
|
|
* request must wait for outstanding read requests to complete.
|
|
|
|
*
|
|
|
|
* Returns 0 if successful, else propagates up the lower layer error.
|
|
|
|
*/
|
|
|
|
int
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_request_await_object(struct i915_request *to,
|
|
|
|
struct drm_i915_gem_object *obj,
|
|
|
|
bool write)
|
2016-09-09 20:11:56 +07:00
|
|
|
{
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
struct dma_fence *excl;
|
|
|
|
int ret = 0;
|
2016-09-09 20:11:56 +07:00
|
|
|
|
|
|
|
if (write) {
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
struct dma_fence **shared;
|
|
|
|
unsigned int count, i;
|
|
|
|
|
2019-08-11 15:06:32 +07:00
|
|
|
ret = dma_resv_get_fences_rcu(obj->base.resv,
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
&excl, &count, &shared);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
for (i = 0; i < count; i++) {
|
2018-02-21 16:56:36 +07:00
|
|
|
ret = i915_request_await_dma_fence(to, shared[i]);
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
if (ret)
|
|
|
|
break;
|
|
|
|
|
|
|
|
dma_fence_put(shared[i]);
|
|
|
|
}
|
|
|
|
|
|
|
|
for (; i < count; i++)
|
|
|
|
dma_fence_put(shared[i]);
|
|
|
|
kfree(shared);
|
2016-09-09 20:11:56 +07:00
|
|
|
} else {
|
2019-08-11 15:06:32 +07:00
|
|
|
excl = dma_resv_get_excl_rcu(obj->base.resv);
|
2016-09-09 20:11:56 +07:00
|
|
|
}
|
|
|
|
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
if (excl) {
|
|
|
|
if (ret == 0)
|
2018-02-21 16:56:36 +07:00
|
|
|
ret = i915_request_await_dma_fence(to, excl);
|
2016-09-09 20:11:56 +07:00
|
|
|
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
dma_fence_put(excl);
|
2016-09-09 20:11:56 +07:00
|
|
|
}
|
|
|
|
|
drm/i915: Move GEM activity tracking into a common struct reservation_object
In preparation to support many distinct timelines, we need to expand the
activity tracking on the GEM object to handle more than just a request
per engine. We already use the struct reservation_object on the dma-buf
to handle many fence contexts, so integrating that into the GEM object
itself is the preferred solution. (For example, we can now share the same
reservation_object between every consumer/producer using this buffer and
skip the manual import/export via dma-buf.)
v2: Reimplement busy-ioctl (by walking the reservation object), postpone
the ABI change for another day. Similarly use the reservation object to
find the last_write request (if active and from i915) for choosing
display CS flips.
Caveats:
* busy-ioctl: busy-ioctl only reports on the native fences, it will not
warn of stalls (in set-domain-ioctl, pread/pwrite etc) if the object is
being rendered to by external fences. It also will not report the same
busy state as wait-ioctl (or polling on the dma-buf) in the same
circumstances. On the plus side, it does retain reporting of which
*i915* engines are engaged with this object.
* non-blocking atomic modesets take a step backwards as the wait for
render completion blocks the ioctl. This is fixed in a subsequent
patch to use a fence instead for awaiting on the rendering, see
"drm/i915: Restore nonblocking awaits for modesetting"
* dynamic array manipulation for shared-fences in reservation is slower
than the previous lockless static assignment (e.g. gem_exec_lut_handle
runtime on ivb goes from 42s to 66s), mainly due to atomic operations
(maintaining the fence refcounts).
* loss of object-level retirement callbacks, emulated by VMA retirement
tracking.
* minor loss of object-level last activity information from debugfs,
could be replaced with per-vma information if desired
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20161028125858.23563-21-chris@chris-wilson.co.uk
2016-10-28 19:58:44 +07:00
|
|
|
return ret;
|
2016-09-09 20:11:56 +07:00
|
|
|
}
|
|
|
|
|
2018-07-06 17:39:43 +07:00
|
|
|
void i915_request_skip(struct i915_request *rq, int error)
|
|
|
|
{
|
|
|
|
void *vaddr = rq->ring->vaddr;
|
|
|
|
u32 head;
|
|
|
|
|
|
|
|
GEM_BUG_ON(!IS_ERR_VALUE((long)error));
|
|
|
|
dma_fence_set_error(&rq->fence, error);
|
|
|
|
|
2019-08-18 06:25:11 +07:00
|
|
|
if (rq->infix == rq->postfix)
|
|
|
|
return;
|
|
|
|
|
2018-07-06 17:39:43 +07:00
|
|
|
/*
|
|
|
|
* As this request likely depends on state from the lost
|
|
|
|
* context, clear out all the user operations leaving the
|
|
|
|
* breadcrumb at the end (so we get the fence notifications).
|
|
|
|
*/
|
|
|
|
head = rq->infix;
|
|
|
|
if (rq->postfix < head) {
|
|
|
|
memset(vaddr + head, 0, rq->ring->size - head);
|
|
|
|
head = 0;
|
|
|
|
}
|
|
|
|
memset(vaddr + head, 0, rq->postfix - head);
|
2019-08-18 06:25:11 +07:00
|
|
|
rq->infix = rq->postfix;
|
2018-07-06 17:39:43 +07:00
|
|
|
}
|
|
|
|
|
2019-03-22 16:23:25 +07:00
|
|
|
static struct i915_request *
|
|
|
|
__i915_request_add_to_timeline(struct i915_request *rq)
|
|
|
|
{
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
struct intel_timeline *timeline = i915_request_timeline(rq);
|
2019-03-22 16:23:25 +07:00
|
|
|
struct i915_request *prev;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Dependency tracking and request ordering along the timeline
|
|
|
|
* is special cased so that we can eliminate redundant ordering
|
|
|
|
* operations while building the request (we know that the timeline
|
|
|
|
* itself is ordered, and here we guarantee it).
|
|
|
|
*
|
|
|
|
* As we know we will need to emit tracking along the timeline,
|
|
|
|
* we embed the hooks into our request struct -- at the cost of
|
|
|
|
* having to have specialised no-allocation interfaces (which will
|
|
|
|
* be beneficial elsewhere).
|
|
|
|
*
|
|
|
|
* A second benefit to open-coding i915_request_await_request is
|
|
|
|
* that we can apply a slight variant of the rules specialised
|
|
|
|
* for timelines that jump between engines (such as virtual engines).
|
|
|
|
* If we consider the case of virtual engine, we must emit a dma-fence
|
|
|
|
* to prevent scheduling of the second request until the first is
|
|
|
|
* complete (to maximise our greedy late load balancing) and this
|
|
|
|
* precludes optimising to use semaphores serialisation of a single
|
|
|
|
* timeline across engines.
|
|
|
|
*/
|
2019-10-04 20:40:00 +07:00
|
|
|
prev = to_request(__i915_active_fence_set(&timeline->last_request,
|
|
|
|
&rq->fence));
|
2019-03-22 16:23:25 +07:00
|
|
|
if (prev && !i915_request_completed(prev)) {
|
|
|
|
if (is_power_of_2(prev->engine->mask | rq->engine->mask))
|
|
|
|
i915_sw_fence_await_sw_fence(&rq->submit,
|
|
|
|
&prev->submit,
|
|
|
|
&rq->submitq);
|
|
|
|
else
|
|
|
|
__i915_sw_fence_await_dma_fence(&rq->submit,
|
|
|
|
&prev->fence,
|
|
|
|
&rq->dmaq);
|
|
|
|
if (rq->engine->schedule)
|
|
|
|
__i915_sched_node_add_dependency(&rq->sched,
|
|
|
|
&prev->sched,
|
|
|
|
&rq->dep,
|
|
|
|
0);
|
|
|
|
}
|
|
|
|
|
|
|
|
list_add_tail(&rq->link, &timeline->requests);
|
|
|
|
|
2019-04-25 03:07:16 +07:00
|
|
|
/*
|
|
|
|
* Make sure that no request gazumped us - if it was allocated after
|
|
|
|
* our i915_request_alloc() and called __i915_request_add() before
|
|
|
|
* us, the timeline will hold its seqno which is later than ours.
|
|
|
|
*/
|
2019-03-22 16:23:25 +07:00
|
|
|
GEM_BUG_ON(timeline->seqno != rq->fence.seqno);
|
|
|
|
|
|
|
|
return prev;
|
|
|
|
}
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/*
|
|
|
|
* NB: This function is not allowed to fail. Doing so would mean the the
|
|
|
|
* request is not being tracked for completion but the work itself is
|
|
|
|
* going to happen on the hardware. This would be a Bad Thing(tm).
|
|
|
|
*/
|
2019-04-25 03:07:16 +07:00
|
|
|
struct i915_request *__i915_request_commit(struct i915_request *rq)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2019-04-25 03:07:16 +07:00
|
|
|
struct intel_engine_cs *engine = rq->engine;
|
|
|
|
struct intel_ring *ring = rq->ring;
|
2017-02-14 18:32:42 +07:00
|
|
|
u32 *cs;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-12-13 22:51:52 +07:00
|
|
|
RQ_TRACE(rq, "\n");
|
2017-01-11 21:08:58 +07:00
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/*
|
|
|
|
* To ensure that this call will not fail, space for its emissions
|
|
|
|
* should already have been reserved in the ring buffer. Let the ring
|
|
|
|
* know that it is time to use that space up.
|
|
|
|
*/
|
2019-04-25 03:07:16 +07:00
|
|
|
GEM_BUG_ON(rq->reserved_space > ring->space);
|
|
|
|
rq->reserved_space = 0;
|
2019-08-16 03:57:09 +07:00
|
|
|
rq->emitted_jiffies = jiffies;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2018-02-07 15:43:50 +07:00
|
|
|
/*
|
|
|
|
* Record the position of the start of the breadcrumb so that
|
2016-07-20 15:21:08 +07:00
|
|
|
* should we detect the updated seqno part-way through the
|
|
|
|
* GPU processing the request, we never over-estimate the
|
2016-08-15 16:48:40 +07:00
|
|
|
* position of the ring's HEAD.
|
2016-07-20 15:21:08 +07:00
|
|
|
*/
|
2019-04-25 03:07:16 +07:00
|
|
|
cs = intel_ring_begin(rq, engine->emit_fini_breadcrumb_dw);
|
2017-02-14 18:32:42 +07:00
|
|
|
GEM_BUG_ON(IS_ERR(cs));
|
2019-04-25 03:07:16 +07:00
|
|
|
rq->postfix = intel_ring_offset(rq, cs);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
return __i915_request_add_to_timeline(rq);
|
2019-08-14 02:07:05 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
void __i915_request_queue(struct i915_request *rq,
|
|
|
|
const struct i915_sched_attr *attr)
|
|
|
|
{
|
2018-02-07 15:43:50 +07:00
|
|
|
/*
|
|
|
|
* Let the backend know a new request has arrived that may need
|
2016-11-15 03:41:01 +07:00
|
|
|
* to adjust the existing execution schedule due to a high priority
|
|
|
|
* request - i.e. we may want to preempt the current request in order
|
|
|
|
* to run a high priority dependency chain *before* we can execute this
|
|
|
|
* request.
|
|
|
|
*
|
|
|
|
* This is called before the request is ready to run so that we can
|
|
|
|
* decide whether to preempt the entire chain so that it is ready to
|
|
|
|
* run at the earliest possible convenience.
|
|
|
|
*/
|
2019-04-25 03:07:16 +07:00
|
|
|
i915_sw_fence_commit(&rq->semaphore);
|
2019-08-14 02:07:05 +07:00
|
|
|
if (attr && rq->engine->schedule)
|
|
|
|
rq->engine->schedule(rq, attr);
|
2019-04-25 03:07:16 +07:00
|
|
|
i915_sw_fence_commit(&rq->submit);
|
|
|
|
}
|
|
|
|
|
|
|
|
void i915_request_add(struct i915_request *rq)
|
|
|
|
{
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
struct intel_timeline * const tl = i915_request_timeline(rq);
|
2019-12-21 23:03:24 +07:00
|
|
|
struct i915_sched_attr attr = {};
|
2019-04-25 03:07:16 +07:00
|
|
|
struct i915_request *prev;
|
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
lockdep_assert_held(&tl->mutex);
|
|
|
|
lockdep_unpin_lock(&tl->mutex, rq->cookie);
|
2019-04-25 03:07:16 +07:00
|
|
|
|
|
|
|
trace_i915_request_add(rq);
|
|
|
|
|
|
|
|
prev = __i915_request_commit(rq);
|
|
|
|
|
2019-12-21 23:03:24 +07:00
|
|
|
if (rq->context->gem_context)
|
|
|
|
attr = rq->context->gem_context->sched;
|
|
|
|
|
2019-08-14 02:07:05 +07:00
|
|
|
/*
|
|
|
|
* Boost actual workloads past semaphores!
|
|
|
|
*
|
|
|
|
* With semaphores we spin on one engine waiting for another,
|
|
|
|
* simply to reduce the latency of starting our work when
|
|
|
|
* the signaler completes. However, if there is any other
|
|
|
|
* work that we could be doing on this engine instead, that
|
|
|
|
* is better utilisation and will reduce the overall duration
|
|
|
|
* of the current work. To avoid PI boosting a semaphore
|
|
|
|
* far in the distance past over useful work, we keep a history
|
|
|
|
* of any semaphore use along our dependency chain.
|
|
|
|
*/
|
|
|
|
if (!(rq->sched.flags & I915_SCHED_HAS_SEMAPHORE_CHAIN))
|
|
|
|
attr.priority |= I915_PRIORITY_NOSEMAPHORE;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Boost priorities to new clients (new request flows).
|
|
|
|
*
|
|
|
|
* Allow interactive/synchronous clients to jump ahead of
|
|
|
|
* the bulk clients. (FQ_CODEL)
|
|
|
|
*/
|
|
|
|
if (list_empty(&rq->sched.signalers_list))
|
|
|
|
attr.priority |= I915_PRIORITY_WAIT;
|
|
|
|
|
2019-08-15 11:20:30 +07:00
|
|
|
local_bh_disable();
|
2019-08-14 02:07:05 +07:00
|
|
|
__i915_request_queue(rq, &attr);
|
2019-08-15 11:20:30 +07:00
|
|
|
local_bh_enable(); /* Kick the execlists tasklet if just scheduled */
|
2019-08-14 02:07:05 +07:00
|
|
|
|
2018-02-07 15:43:49 +07:00
|
|
|
/*
|
|
|
|
* In typical scenarios, we do not expect the previous request on
|
|
|
|
* the timeline to be still tracked by timeline->last_request if it
|
|
|
|
* has been completed. If the completed request is still here, that
|
|
|
|
* implies that request retirement is a long way behind submission,
|
|
|
|
* suggesting that we haven't been retiring frequently enough from
|
|
|
|
* the combination of retire-before-alloc, waiters and the background
|
|
|
|
* retirement worker. So if the last request on this timeline was
|
|
|
|
* already completed, do a catch up pass, flushing the retirement queue
|
|
|
|
* up to this client. Since we have now moved the heaviest operations
|
|
|
|
* during retirement onto secondary workers, such as freeing objects
|
|
|
|
* or contexts, retiring a bunch of requests is mostly list management
|
|
|
|
* (and cache misses), and so we should not be overly penalizing this
|
|
|
|
* client by performing excess work, though we may still performing
|
|
|
|
* work on behalf of others -- but instead we should benefit from
|
|
|
|
* improved resource management. (Well, that's the theory at least.)
|
|
|
|
*/
|
drm/i915: Mark i915_request.timeline as a volatile, rcu pointer
The request->timeline is only valid until the request is retired (i.e.
before it is completed). Upon retiring the request, the context may be
unpinned and freed, and along with it the timeline may be freed. We
therefore need to be very careful when chasing rq->timeline that the
pointer does not disappear beneath us. The vast majority of users are in
a protected context, either during request construction or retirement,
where the timeline->mutex is held and the timeline cannot disappear. It
is those few off the beaten path (where we access a second timeline) that
need extra scrutiny -- to be added in the next patch after first adding
the warnings about dangerous access.
One complication, where we cannot use the timeline->mutex itself, is
during request submission onto hardware (under spinlocks). Here, we want
to check on the timeline to finalize the breadcrumb, and so we need to
impose a second rule to ensure that the request->timeline is indeed
valid. As we are submitting the request, it's context and timeline must
be pinned, as it will be used by the hardware. Since it is pinned, we
know the request->timeline must still be valid, and we cannot submit the
idle barrier until after we release the engine->active.lock, ergo while
submitting and holding that spinlock, a second thread cannot release the
timeline.
v2: Don't be lazy inside selftests; hold the timeline->mutex for as long
as we need it, and tidy up acquiring the timeline with a bit of
refactoring (i915_active_add_request)
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190919111912.21631-1-chris@chris-wilson.co.uk
2019-09-19 18:19:10 +07:00
|
|
|
if (prev &&
|
|
|
|
i915_request_completed(prev) &&
|
|
|
|
rcu_access_pointer(prev->timeline) == tl)
|
2018-02-21 16:56:36 +07:00
|
|
|
i915_request_retire_upto(prev);
|
2019-03-01 18:05:44 +07:00
|
|
|
|
2019-08-16 03:57:09 +07:00
|
|
|
mutex_unlock(&tl->mutex);
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned long local_clock_us(unsigned int *cpu)
|
|
|
|
{
|
|
|
|
unsigned long t;
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* Cheaply and approximately convert from nanoseconds to microseconds.
|
2016-07-20 15:21:08 +07:00
|
|
|
* The result and subsequent calculations are also defined in the same
|
|
|
|
* approximate microseconds units. The principal source of timing
|
|
|
|
* error here is from the simple truncation.
|
|
|
|
*
|
|
|
|
* Note that local_clock() is only defined wrt to the current CPU;
|
|
|
|
* the comparisons are no longer valid if we switch CPUs. Instead of
|
|
|
|
* blocking preemption for the entire busywait, we can detect the CPU
|
|
|
|
* switch and use that as indicator of system load and a reason to
|
|
|
|
* stop busywaiting, see busywait_stop().
|
|
|
|
*/
|
|
|
|
*cpu = get_cpu();
|
|
|
|
t = local_clock() >> 10;
|
|
|
|
put_cpu();
|
|
|
|
|
|
|
|
return t;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool busywait_stop(unsigned long timeout, unsigned int cpu)
|
|
|
|
{
|
|
|
|
unsigned int this_cpu;
|
|
|
|
|
|
|
|
if (time_after(local_clock_us(&this_cpu), timeout))
|
|
|
|
return true;
|
|
|
|
|
|
|
|
return this_cpu != cpu;
|
|
|
|
}
|
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
static bool __i915_spin_request(const struct i915_request * const rq,
|
|
|
|
int state, unsigned long timeout_us)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
unsigned int cpu;
|
2017-09-22 19:03:33 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Only wait for the request if we know it is likely to complete.
|
|
|
|
*
|
|
|
|
* We don't track the timestamps around requests, nor the average
|
|
|
|
* request length, so we do not have a good indicator that this
|
|
|
|
* request will complete within the timeout. What we do know is the
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
* order in which requests are executed by the context and so we can
|
|
|
|
* tell if the request has been started. If the request is not even
|
|
|
|
* running yet, it is a fair assumption that it will not complete
|
|
|
|
* within our relatively short timeout.
|
2017-09-22 19:03:33 +07:00
|
|
|
*/
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
if (!i915_request_is_running(rq))
|
2017-09-22 19:03:33 +07:00
|
|
|
return false;
|
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
/*
|
|
|
|
* When waiting for high frequency requests, e.g. during synchronous
|
2016-07-20 15:21:08 +07:00
|
|
|
* rendering split between the CPU and GPU, the finite amount of time
|
|
|
|
* required to set up the irq and wait upon it limits the response
|
|
|
|
* rate. By busywaiting on the request completion for a short while we
|
|
|
|
* can service the high frequency waits as quick as possible. However,
|
|
|
|
* if it is a slow request, we want to sleep as quickly as possible.
|
|
|
|
* The tradeoff between waiting and sleeping is roughly the time it
|
|
|
|
* takes to sleep on a request, on the order of a microsecond.
|
|
|
|
*/
|
|
|
|
|
|
|
|
timeout_us += local_clock_us(&cpu);
|
|
|
|
do {
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
if (i915_request_completed(rq))
|
|
|
|
return true;
|
2017-02-17 22:13:01 +07:00
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
if (signal_pending_state(state, current))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (busywait_stop(timeout_us, cpu))
|
|
|
|
break;
|
|
|
|
|
2016-10-25 16:03:14 +07:00
|
|
|
cpu_relax();
|
2016-07-20 15:21:08 +07:00
|
|
|
} while (!need_resched());
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
struct request_wait {
|
|
|
|
struct dma_fence_cb cb;
|
|
|
|
struct task_struct *tsk;
|
|
|
|
};
|
|
|
|
|
|
|
|
static void request_wait_wake(struct dma_fence *fence, struct dma_fence_cb *cb)
|
|
|
|
{
|
|
|
|
struct request_wait *wait = container_of(cb, typeof(*wait), cb);
|
|
|
|
|
|
|
|
wake_up_process(wait->tsk);
|
|
|
|
}
|
|
|
|
|
2016-07-20 15:21:08 +07:00
|
|
|
/**
|
2018-02-23 00:24:05 +07:00
|
|
|
* i915_request_wait - wait until execution of request has finished
|
2018-02-21 16:56:36 +07:00
|
|
|
* @rq: the request to wait upon
|
2016-09-09 20:11:49 +07:00
|
|
|
* @flags: how to wait
|
2016-10-28 19:58:27 +07:00
|
|
|
* @timeout: how long to wait in jiffies
|
|
|
|
*
|
2018-02-23 00:24:05 +07:00
|
|
|
* i915_request_wait() waits for the request to be completed, for a
|
2016-10-28 19:58:27 +07:00
|
|
|
* maximum of @timeout jiffies (with MAX_SCHEDULE_TIMEOUT implying an
|
|
|
|
* unbounded wait).
|
2016-07-20 15:21:08 +07:00
|
|
|
*
|
2016-10-28 19:58:27 +07:00
|
|
|
* Returns the remaining time (in jiffies) if the request completed, which may
|
|
|
|
* be zero or -ETIME if the request is unfinished after the timeout expires.
|
|
|
|
* May return -EINTR is called with I915_WAIT_INTERRUPTIBLE and a signal is
|
|
|
|
* pending before the request completes.
|
2016-07-20 15:21:08 +07:00
|
|
|
*/
|
2018-02-21 16:56:36 +07:00
|
|
|
long i915_request_wait(struct i915_request *rq,
|
2016-10-28 19:58:27 +07:00
|
|
|
unsigned int flags,
|
|
|
|
long timeout)
|
2016-07-20 15:21:08 +07:00
|
|
|
{
|
2016-09-09 20:11:49 +07:00
|
|
|
const int state = flags & I915_WAIT_INTERRUPTIBLE ?
|
|
|
|
TASK_INTERRUPTIBLE : TASK_UNINTERRUPTIBLE;
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
struct request_wait wait;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
|
|
|
might_sleep();
|
2016-10-28 19:58:27 +07:00
|
|
|
GEM_BUG_ON(timeout < 0);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-06-14 18:10:53 +07:00
|
|
|
if (dma_fence_is_signaled(&rq->fence))
|
2016-10-28 19:58:27 +07:00
|
|
|
return timeout;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2016-10-28 19:58:27 +07:00
|
|
|
if (!timeout)
|
|
|
|
return -ETIME;
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2018-02-21 16:56:36 +07:00
|
|
|
trace_i915_request_wait_begin(rq, flags);
|
2019-06-14 14:09:46 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We must never wait on the GPU while holding a lock as we
|
|
|
|
* may need to perform a GPU reset. So while we don't need to
|
|
|
|
* serialise wait/reset with an explicit lock, we do want
|
|
|
|
* lockdep to detect potential dependency cycles.
|
|
|
|
*/
|
2019-07-13 02:29:53 +07:00
|
|
|
mutex_acquire(&rq->engine->gt->reset.mutex.dep_map, 0, 0, _THIS_IP_);
|
2016-10-28 19:58:48 +07:00
|
|
|
|
2019-04-20 01:26:25 +07:00
|
|
|
/*
|
|
|
|
* Optimistic spin before touching IRQs.
|
|
|
|
*
|
|
|
|
* We may use a rather large value here to offset the penalty of
|
|
|
|
* switching away from the active task. Frequently, the client will
|
|
|
|
* wait upon an old swapbuffer to throttle itself to remain within a
|
|
|
|
* frame of the gpu. If the client is running in lockstep with the gpu,
|
|
|
|
* then it should not be waiting long at all, and a sleep now will incur
|
|
|
|
* extra scheduler latency in producing the next frame. To try to
|
|
|
|
* avoid adding the cost of enabling/disabling the interrupt to the
|
|
|
|
* short wait, we first spin to see if the request would have completed
|
|
|
|
* in the time taken to setup the interrupt.
|
|
|
|
*
|
|
|
|
* We need upto 5us to enable the irq, and upto 20us to hide the
|
|
|
|
* scheduler latency of a context switch, ignoring the secondary
|
|
|
|
* impacts from a context switch such as cache eviction.
|
|
|
|
*
|
|
|
|
* The scheme used for low-latency IO is called "hybrid interrupt
|
|
|
|
* polling". The suggestion there is to sleep until just before you
|
|
|
|
* expect to be woken by the device interrupt and then poll for its
|
|
|
|
* completion. That requires having a good predictor for the request
|
|
|
|
* duration, which we currently lack.
|
|
|
|
*/
|
2019-10-25 20:59:42 +07:00
|
|
|
if (IS_ACTIVE(CONFIG_DRM_I915_SPIN_REQUEST) &&
|
2019-06-14 18:10:53 +07:00
|
|
|
__i915_spin_request(rq, state, CONFIG_DRM_I915_SPIN_REQUEST)) {
|
|
|
|
dma_fence_signal(&rq->fence);
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
goto out;
|
2019-06-14 18:10:53 +07:00
|
|
|
}
|
2017-02-23 14:44:12 +07:00
|
|
|
|
2019-02-13 16:25:04 +07:00
|
|
|
/*
|
|
|
|
* This client is about to stall waiting for the GPU. In many cases
|
|
|
|
* this is undesirable and limits the throughput of the system, as
|
|
|
|
* many clients cannot continue processing user input/output whilst
|
|
|
|
* blocked. RPS autotuning may take tens of milliseconds to respond
|
|
|
|
* to the GPU load and thus incurs additional latency for the client.
|
|
|
|
* We can circumvent that by promoting the GPU frequency to maximum
|
|
|
|
* before we sleep. This makes the GPU throttle up much more quickly
|
|
|
|
* (good for benchmarks and user experience, e.g. window animations),
|
|
|
|
* but at a cost of spending more power processing the workload
|
|
|
|
* (bad for battery).
|
|
|
|
*/
|
|
|
|
if (flags & I915_WAIT_PRIORITY) {
|
|
|
|
if (!i915_request_started(rq) && INTEL_GEN(rq->i915) >= 6)
|
2019-10-25 04:16:41 +07:00
|
|
|
intel_rps_boost(rq);
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
i915_schedule_bump_priority(rq, I915_PRIORITY_WAIT);
|
2019-02-13 16:25:04 +07:00
|
|
|
}
|
2016-10-28 19:58:48 +07:00
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
wait.tsk = current;
|
|
|
|
if (dma_fence_add_callback(&rq->fence, &wait.cb, request_wait_wake))
|
|
|
|
goto out;
|
2016-10-28 19:58:48 +07:00
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
for (;;) {
|
|
|
|
set_current_state(state);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
2019-06-19 18:23:37 +07:00
|
|
|
if (i915_request_completed(rq)) {
|
|
|
|
dma_fence_signal(&rq->fence);
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
break;
|
2019-06-19 18:23:37 +07:00
|
|
|
}
|
2016-07-20 15:21:08 +07:00
|
|
|
|
|
|
|
if (signal_pending_state(state, current)) {
|
2016-10-28 19:58:27 +07:00
|
|
|
timeout = -ERESTARTSYS;
|
2016-07-20 15:21:08 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2016-10-28 19:58:27 +07:00
|
|
|
if (!timeout) {
|
|
|
|
timeout = -ETIME;
|
2016-07-20 15:21:08 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2019-10-15 20:26:06 +07:00
|
|
|
intel_engine_flush_submission(rq->engine);
|
2016-10-28 19:58:27 +07:00
|
|
|
timeout = io_schedule_timeout(timeout);
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
2017-02-23 14:44:19 +07:00
|
|
|
__set_current_state(TASK_RUNNING);
|
2016-07-20 15:21:08 +07:00
|
|
|
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
dma_fence_remove_callback(&rq->fence, &wait.cb);
|
|
|
|
|
|
|
|
out:
|
2019-09-19 23:09:40 +07:00
|
|
|
mutex_release(&rq->engine->gt->reset.mutex.dep_map, _THIS_IP_);
|
drm/i915: Replace global breadcrumbs with per-context interrupt tracking
A few years ago, see commit 688e6c725816 ("drm/i915: Slaughter the
thundering i915_wait_request herd"), the issue of handling multiple
clients waiting in parallel was brought to our attention. The
requirement was that every client should be woken immediately upon its
request being signaled, without incurring any cpu overhead.
To handle certain fragility of our hw meant that we could not do a
simple check inside the irq handler (some generations required almost
unbounded delays before we could be sure of seqno coherency) and so
request completion checking required delegation.
Before commit 688e6c725816, the solution was simple. Every client
waiting on a request would be woken on every interrupt and each would do
a heavyweight check to see if their request was complete. Commit
688e6c725816 introduced an rbtree so that only the earliest waiter on
the global timeline would woken, and would wake the next and so on.
(Along with various complications to handle requests being reordered
along the global timeline, and also a requirement for kthread to provide
a delegate for fence signaling that had no process context.)
The global rbtree depends on knowing the execution timeline (and global
seqno). Without knowing that order, we must instead check all contexts
queued to the HW to see which may have advanced. We trim that list by
only checking queued contexts that are being waited on, but still we
keep a list of all active contexts and their active signalers that we
inspect from inside the irq handler. By moving the waiters onto the fence
signal list, we can combine the client wakeup with the dma_fence
signaling (a dramatic reduction in complexity, but does require the HW
being coherent, the seqno must be visible from the cpu before the
interrupt is raised - we keep a timer backup just in case).
Having previously fixed all the issues with irq-seqno serialisation (by
inserting delays onto the GPU after each request instead of random delays
on the CPU after each interrupt), we can rely on the seqno state to
perfom direct wakeups from the interrupt handler. This allows us to
preserve our single context switch behaviour of the current routine,
with the only downside that we lose the RT priority sorting of wakeups.
In general, direct wakeup latency of multiple clients is about the same
(about 10% better in most cases) with a reduction in total CPU time spent
in the waiter (about 20-50% depending on gen). Average herd behaviour is
improved, but at the cost of not delegating wakeups on task_prio.
v2: Capture fence signaling state for error state and add comments to
warm even the most cold of hearts.
v3: Check if the request is still active before busywaiting
v4: Reduce the amount of pointer misdirection with list_for_each_safe
and using a local i915_request variable inside the loops
v5: Add a missing pluralisation to a purely informative selftest message.
References: 688e6c725816 ("drm/i915: Slaughter the thundering i915_wait_request herd")
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20190129205230.19056-2-chris@chris-wilson.co.uk
2019-01-30 03:52:29 +07:00
|
|
|
trace_i915_request_wait_end(rq);
|
2016-10-28 19:58:27 +07:00
|
|
|
return timeout;
|
2016-07-20 15:21:08 +07:00
|
|
|
}
|
2016-08-04 13:52:42 +07:00
|
|
|
|
2017-02-14 00:15:21 +07:00
|
|
|
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
|
|
|
|
#include "selftests/mock_request.c"
|
2018-02-21 16:56:36 +07:00
|
|
|
#include "selftests/i915_request.c"
|
2017-02-14 00:15:21 +07:00
|
|
|
#endif
|
2019-02-28 17:20:33 +07:00
|
|
|
|
2019-03-06 04:38:30 +07:00
|
|
|
static void i915_global_request_shrink(void)
|
|
|
|
{
|
|
|
|
kmem_cache_shrink(global.slab_dependencies);
|
|
|
|
kmem_cache_shrink(global.slab_execute_cbs);
|
|
|
|
kmem_cache_shrink(global.slab_requests);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void i915_global_request_exit(void)
|
|
|
|
{
|
|
|
|
kmem_cache_destroy(global.slab_dependencies);
|
|
|
|
kmem_cache_destroy(global.slab_execute_cbs);
|
|
|
|
kmem_cache_destroy(global.slab_requests);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct i915_global_request global = { {
|
|
|
|
.shrink = i915_global_request_shrink,
|
|
|
|
.exit = i915_global_request_exit,
|
|
|
|
} };
|
|
|
|
|
2019-02-28 17:20:33 +07:00
|
|
|
int __init i915_global_request_init(void)
|
|
|
|
{
|
drm/i915: Use a ctor for TYPESAFE_BY_RCU i915_request
As we start peeking into requests for longer and longer, e.g.
incorporating use of spinlocks when only protected by an
rcu_read_lock(), we need to be careful in how we reset the request when
recycling and need to preserve any barriers that may still be in use as
the request is reset for reuse.
Quoting Linus Torvalds:
> If there is refcounting going on then why use SLAB_TYPESAFE_BY_RCU?
.. because the object can be accessed (by RCU) after the refcount has
gone down to zero, and the thing has been released.
That's the whole and only point of SLAB_TYPESAFE_BY_RCU.
That flag basically says:
"I may end up accessing this object *after* it has been free'd,
because there may be RCU lookups in flight"
This has nothing to do with constructors. It's ok if the object gets
reused as an object of the same type and does *not* get
re-initialized, because we're perfectly fine seeing old stale data.
What it guarantees is that the slab isn't shared with any other kind
of object, _and_ that the underlying pages are free'd after an RCU
quiescent period (so the pages aren't shared with another kind of
object either during an RCU walk).
And it doesn't necessarily have to have a constructor, because the
thing that a RCU walk will care about is
(a) guaranteed to be an object that *has* been on some RCU list (so
it's not a "new" object)
(b) the RCU walk needs to have logic to verify that it's still the
*same* object and hasn't been re-used as something else.
In contrast, a SLAB_TYPESAFE_BY_RCU memory gets free'd and re-used
immediately, but because it gets reused as the same kind of object,
the RCU walker can "know" what parts have meaning for re-use, in a way
it couidn't if the re-use was random.
That said, it *is* subtle, and people should be careful.
> So the re-use might initialize the fields lazily, not necessarily using a ctor.
If you have a well-defined refcount, and use "atomic_inc_not_zero()"
to guard the speculative RCU access section, and use
"atomic_dec_and_test()" in the freeing section, then you should be
safe wrt new allocations.
If you have a completely new allocation that has "random stale
content", you know that it cannot be on the RCU list, so there is no
speculative access that can ever see that random content.
So the only case you need to worry about is a re-use allocation, and
you know that the refcount will start out as zero even if you don't
have a constructor.
So you can think of the refcount itself as always having a zero
constructor, *BUT* you need to be careful with ordering.
In particular, whoever does the allocation needs to then set the
refcount to a non-zero value *after* it has initialized all the other
fields. And in particular, it needs to make sure that it uses the
proper memory ordering to do so.
NOTE! One thing to be very worried about is that re-initializing
whatever RCU lists means that now the RCU walker may be walking on the
wrong list so the walker may do the right thing for this particular
entry, but it may miss walking *other* entries. So then you can get
spurious lookup failures, because the RCU walker never walked all the
way to the end of the right list. That ends up being a much more
subtle bug.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191122094924.629690-1-chris@chris-wilson.co.uk
2019-11-22 16:49:24 +07:00
|
|
|
global.slab_requests =
|
|
|
|
kmem_cache_create("i915_request",
|
|
|
|
sizeof(struct i915_request),
|
|
|
|
__alignof__(struct i915_request),
|
|
|
|
SLAB_HWCACHE_ALIGN |
|
|
|
|
SLAB_RECLAIM_ACCOUNT |
|
|
|
|
SLAB_TYPESAFE_BY_RCU,
|
|
|
|
__i915_request_ctor);
|
2019-02-28 17:20:33 +07:00
|
|
|
if (!global.slab_requests)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
global.slab_execute_cbs = KMEM_CACHE(execute_cb,
|
|
|
|
SLAB_HWCACHE_ALIGN |
|
|
|
|
SLAB_RECLAIM_ACCOUNT |
|
|
|
|
SLAB_TYPESAFE_BY_RCU);
|
|
|
|
if (!global.slab_execute_cbs)
|
|
|
|
goto err_requests;
|
|
|
|
|
2019-02-28 17:20:33 +07:00
|
|
|
global.slab_dependencies = KMEM_CACHE(i915_dependency,
|
|
|
|
SLAB_HWCACHE_ALIGN |
|
|
|
|
SLAB_RECLAIM_ACCOUNT);
|
|
|
|
if (!global.slab_dependencies)
|
2019-03-02 00:09:00 +07:00
|
|
|
goto err_execute_cbs;
|
2019-02-28 17:20:33 +07:00
|
|
|
|
2019-03-06 04:38:30 +07:00
|
|
|
i915_global_register(&global.base);
|
2019-02-28 17:20:33 +07:00
|
|
|
return 0;
|
|
|
|
|
2019-03-02 00:09:00 +07:00
|
|
|
err_execute_cbs:
|
|
|
|
kmem_cache_destroy(global.slab_execute_cbs);
|
2019-02-28 17:20:33 +07:00
|
|
|
err_requests:
|
|
|
|
kmem_cache_destroy(global.slab_requests);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|