Commit Graph

50 Commits

Author SHA1 Message Date
Hang Yuan
1ca20f33df drm/i915/gvt: add hotplug emulation
Add function to emulate hotplug interrupt for SKL/KBL platforms

Signed-off-by: Hang Yuan <hang.yuan@linux.intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2019-01-31 11:26:55 +08:00
Zhenyu Wang
9bdb073464 drm/i915/gvt: Change KVMGT as self load module
This trys to make 'kvmgt' module as self loadable instead of loading
by i915/gvt device model. So hypervisor specific module could be
stand-alone, e.g only after loading hypervisor specific module, GVT
feature could be enabled via specific hypervisor interface, e.g VFIO/mdev.

So this trys to use hypervisor module register/unregister interface
for that. Hypervisor module needs to take care of module reference
itself when working for hypervisor interface, e.g for VFIO/mdev,
hypervisor module would reference counting mdev when open and release.

This makes 'kvmgt' module really split from GVT device model. User
needs to load 'kvmgt' to enable VFIO/mdev interface.

v6:
- remove unused variable

v5:
- put module reference in register error path

v4:
- fix checkpatch warning

v3:
- Fix module reference handling for device open and release. Unused
  mdev devices would be cleaned up in device unregister when module unload.

v2:
- Fix kvmgt order after i915 for built-in case

Cc: "Yuan, Hang" <hang.yuan@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: "He, Min" <min.he@intel.com>
Reviewed-by: Yuan, Hang <hang.yuan@intel.com>
Acked-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2019-01-02 15:51:26 +08:00
Zhenyu Wang
a2b8419a9e drm/i915/gvt: remove unused parameter for hypervisor's host_exit call
The parameter 'void *gvt' is not used and required for hypervisor's
exit call. Even for non-merged Xen hypervisor support. So just remove it.

Reviewed-by: Yuan, Hang <hang.yuan@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-12-10 15:12:19 +08:00
Peng Hao
77453c9f2b drm/i915/gvt: fix a typo: "registeration" -> "registration".
Fix a typo in the error message reporting.

Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-12-07 12:00:58 +08:00
Dave Airlie
b1c1566822 Merge tag 'drm-intel-next-2018-09-06-2' of git://anongit.freedesktop.org/drm/drm-intel into drm-next
Merge tag 'gvt-next-2018-09-04'
drm-intel-next-2018-09-06-1:
UAPI Changes:
- GGTT coherency GETPARAM: GGTT has turned out to be non-coherent for some
  platforms, which we've failed to communicate to userspace so far. SNA was
  modified to do extra flushing on non-coherent GGTT access, while Mesa will
  mitigate by always requiring WC mapping (which is non-coherent anyway).
- Neuter Resource Streamer uAPI: There never really were users for the feature,
  so neuter it while keeping the interface bits for compatibility. This is a
  long due item from past.

Cross-subsystem Changes:
- Backmerge of branch drm-next-4.19 for DP_DPCD_REV_14 changes

Core Changes:
- None

Driver Changes:

- A load of Icelake (ICL) enabling patches (Paulo, Manasi)
- Enabled full PPGTT for IVB,VLV and HSW (Chris)
- Bugzilla #107113: Distribute DDB based on display resolutions (Mahesh)
- Bugzillas #100023,#107476,#94921: Support limited range DP displays (Jani)
- Bugzilla #107503: Increase LSPCON timeout (Fredrik)
- Avoid boosting GPU due to an occasional stall in interactive workloads (Chris)
- Apply GGTT coherency W/A only for affected systems instead of all (Chris)
- Fix for infinite link training loop for faulty USB-C MST hubs (Nathan)
- Keep KMS functional on Gen4 and earlier when GPU is wedged (Chris)
- Stop holding ppGTT reference from closed VMAs (Chris)
- Clear error registers after error capture (Lionel)
- Various Icelake fixes (Anusha, Jyoti, Ville, Tvrtko)
- Add missing Coffeelake (CFL) PCI IDs (Rodrigo)
- Flush execlists tasklet directly from reset-finish (Chris)
- Fix LPE audio runtime PM (Chris)
- Fix detection of out of range surface positions (GLK/CNL) (Ville)
- Remove wait-for-idle for PSR2 (Dhinakaran)
- Power down existing display hardware resources when display is disabled (Chris)
- Don't allow runtime power management if RC6 doesn't exist (Chris)
- Add debugging checks for runtime power management paths (Imre)
- Increase symmetry in display power init/fini paths (Imre)
- Isolate GVT specific macros from i915_reg.h (Lucas)
- Increase symmetry in power management enable/disable paths (Chris)
- Increase IP disable timeout to 100 ms to avoid DRM_ERROR (Imre)
- Fix memory leak from HDMI HDCP write function (Brian, Rodrigo)
- Reject Y/Yf tiling on interlaced modes (Ville)
- Use a cached mapping for the physical HWS on older gens (Chris)
- Force slow path of writing relocations to buffer if unable to write to userspace (Chris)
- Do a full device reset after being wedged (Chris)
- Keep forcewake counts over reset (in case of debugfs user) (Imre, Chris)
- Avoid false-positive errors from power wells during init (Imre)
- Reset engines forcibly in exchange of declaring whole device wedged (Mika)
- Reduce context HW ID lifetime in preparation for Icelake (Chris)
- Attempt to recover from module load failures (Chris)
- Keep select interrupts over a reset to avoid missing/losing them (Chris)
- GuC submission backend improvements (Jakub)
- Terminate context images with BB_END (Chris, Lionel)
- Make GCC evaluate GGTT view struct size assertions again (Ville)
- Add selftest to exercise suspend/hibernate code-paths for GEM (Chris)
- Use a full emulation of a user ppgtt context in selftests (Chris)
- Exercise resetting in the middle of a wait-on-fence in selftests (Chris)
- Fix coherency issues on selftests for Baytrail (Chris)
- Various other GEM fixes / self-test updates (Chris, Matt)
- GuC doorbell self-tests (Daniele)
- PSR mode control through debugfs for IGTs (Maarten)
- Degrade expected WM latency errors to DRM_DEBUG_KMS (Chris)
- Cope with errors better in MST link training (Dhinakaran)
- Fix WARN on KBL external displays (Azhar)
- Power well code cleanups (Imre)
- Fixes to PSR debugging (Dhinakaran)
- Make forcewake errors louder for easier catching in CI (WARNs) (Chris)
- Fortify tiling code against programmer errors (Chris)
- Bunch of fixes for CI exposed corner cases (multiple authors, mostly Chris)

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180907105446.GA22860@jlahtine-desk.ger.corp.intel.com
2018-09-11 11:53:12 +10:00
Hang Yuan
3fd34ac02a drm/i915/gvt: fix cleanup sequence in intel_gvt_clean_device
Create one vGPU and then unbind IGD device from i915 driver. The following
oops will happen. This patch will free vgpu resource first and then gvt
resource to remove these oops.

BUG: unable to handle kernel NULL pointer dereference at       00000000000000a8
  PGD 80000003c9d2c067 P4D 80000003c9d2c067 PUD 3c817c067 P      MD 0
  Oops: 0002 [#1] SMP PTI
  RIP: 0010:down_write+0x1b/0x40
Call Trace:
  debugfs_remove_recursive+0x46/0x1a0
  intel_gvt_debugfs_remove_vgpu+0x15/0x30 [i915]
  intel_gvt_destroy_vgpu+0x2d/0xf0 [i915]
  intel_vgpu_remove+0x2c/0x30 [kvmgt]
  mdev_device_remove_ops+0x23/0x50 [mdev]
  mdev_device_remove+0xdb/0x190 [mdev]
  mdev_device_remove+0x190/0x190 [mdev]
  device_for_each_child+0x47/0x90
  mdev_unregister_device+0xd5/0x120 [mdev]
  intel_gvt_clean_device+0x91/0x120 [i915]
  i915_driver_unload+0x9d/0x120 [i915]
  i915_pci_remove+0x15/0x20 [i915]
  pci_device_remove+0x3b/0xc0
  device_release_driver_internal+0x157/0x230
  unbind_store+0xfc/0x150
  kernfs_fop_write+0x10f/0x180
  __vfs_write+0x36/0x180
  ? common_file_perm+0x41/0x130
  ? _cond_resched+0x16/0x40
  vfs_write+0xb3/0x1a0
  ksys_write+0x52/0xc0
  do_syscall_64+0x55/0x100
  entry_SYSCALL_64_after_hwframe+0x44/0xa9

BUG: unable to handle kernel NULL pointer dereference at 0      000000000000038
  PGD 8000000405bce067 P4D 8000000405bce067 PUD 405bcd067 PM      D 0
  Oops: 0000 [#1] SMP PTI
  RIP: 0010:hrtimer_active+0x5/0x40
Call Trace:
  hrtimer_try_to_cancel+0x25/0x120
  ? tbs_sched_clean_vgpu+0x1f/0x50 [i915]
  hrtimer_cancel+0x15/0x20
  intel_gvt_destroy_vgpu+0x4c/0xf0 [i915]
  intel_vgpu_remove+0x2c/0x30 [kvmgt]
  mdev_device_remove_ops+0x23/0x50 [mdev]
  mdev_device_remove+0xdb/0x190 [mdev]
  ? mdev_device_remove+0x190/0x190 [mdev]
  device_for_each_child+0x47/0x90
  mdev_unregister_device+0xd5/0x120 [mdev]
  intel_gvt_clean_device+0x89/0x120 [i915]
  i915_driver_unload+0x9d/0x120 [i915]
  i915_pci_remove+0x15/0x20 [i915]
  pci_device_remove+0x3b/0xc0
  device_release_driver_internal+0x157/0x230
  unbind_store+0xfc/0x150
  kernfs_fop_write+0x10f/0x180
  __vfs_write+0x36/0x180
  ? common_file_perm+0x41/0x130
  ? _cond_resched+0x16/0x40
  vfs_write+0xb3/0x1a0
  ksys_write+0x52/0xc0
  do_syscall_64+0x55/0x100
  entry_SYSCALL_64_after_hwframe+0x44/0xa9

Fixes: bc7b0be316ae("drm/i915/gvt: Add basic debugfs infrastructure")
Fixes: afe04fbe6c52("drm/i915/gvt: create an idle vGPU")
Signed-off-by: Hang Yuan <hang.yuan@linux.intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-08-14 15:25:32 +08:00
Hang Yuan
f9090d4c22 drm/i915/gvt: free workload in vgpu release
Some workloads may be prepared in vgpu's queue but not be scheduled
to run yet. If vgpu is released at this time, they will not be freed
in workload complete callback and so need to be freed in vgpu release
operation.

Add new vgpu_release operation in gvt_ops to stop vgpu and release
runtime resources. gvt_ops vgpu_deactivate operation will only stop
vgpu.

v2: add new gvt ops to clean vgpu running status (Xiong Zhang)

Signed-off-by: Hang Yuan <hang.yuan@linux.intel.com>
Reviewed-by: Xiong Zhang <xiong.y.zhang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-08-13 12:32:57 +08:00
Zhenyu Wang
a752b070a6 drm/i915/gvt: Fix function comment doc errors
Caught by W=1 to fix left wrong function comment doc.

Reviewed-by: Hang Yuan <hang.yuan@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-08-07 10:39:53 +08:00
Hang Yuan
279ce5d117 drm/i915/gvt: declare gvt as i915's soft dependency
This helps initramfs builder and other tools to know the full dependencies
of i915 and have gvt module loaded with i915.

v2: add condition and change to pre-dependency (Chris)
v3: move declaration to gvt.c. (Chris)
v4: remove xengt (Zhenyu)

Signed-off-by: Hang Yuan <hang.yuan@linux.intel.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-07-10 11:13:11 +08:00
Colin Xu
6fa6616b8f drm/i915/gvt: Enable device info initialization for BXT.
Initialize BXT device info as SKL/KBL.

v2: All supported platforms share the same device configuration.
    Remove the platform check by now and let is_supported_device()
    be the gate keeper.

Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-06-13 10:57:29 +08:00
Colin Xu
9a512e23f1 drm/i915/gvt: Use sched_lock to protect gvt scheduler logic.
The scheduler lock(gvt->sched_lock) is used to protect gvt
scheduler logic, including the gvt scheduler structure(gvt->scheduler
and per vgpu schedule data(vgpu->sched_data, vgpu->sched_ctl).

v9:
  - Change commit author since the patches are improved a lot compared
    with original version.
    Original author: Pei Zhang <pei.zhang@intel.com>
  - Rebase to latest gvt-staging.
v8:
  - Correct coding wqstyle.
  - Rebase to latest gvt-staging.
v7:
  - Remove gtt_lock since already proteced by gvt_lock and vgpu_lock.
v6:
  - Rebase to latest gvt-staging.
v5:
  - Rebase to latest gvt-staging.
v4:
  - Rebase to latest gvt-staging.
v3: update to latest code base

Signed-off-by: Pei Zhang <pei.zhang@intel.com>
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-05-18 12:39:26 +08:00
Colin Xu
f25a49ab8a drm/i915/gvt: Use vgpu_lock to protect per vgpu access
The patch set splits out 2 small locks from the original big gvt lock:
  - vgpu_lock protects per-vGPU data and logic, especially the vGPU
    trap emulation path.
  - sched_lock protects gvt scheudler structure, context schedule logic
    and vGPU's schedule data.

Use vgpu_lock to replace the gvt big lock. By doing this, the
mmio read/write trap path, vgpu virtual event emulation and other
vgpu related process, would be protected under per vgpu_lock.

v9:
  - Change commit author since the patches are improved a lot compared
    with original version.
    Original author: Pei Zhang <pei.zhang@intel.com>
  - Rebase to latest gvt-staging.
v8:
  - Correct coding and comment style.
  - Rebase to latest gvt-staging.
v7:
  - Remove gtt_lock since already proteced by gvt_lock and vgpu_lock.
  - Fix a typo in intel_gvt_deactivate_vgpu, unlock the wrong lock.
v6:
  - Rebase to latest gvt-staging.
v5:
  - Rebase to latest gvt-staging.
  - intel_vgpu_page_track_handler should use vgpu_lock.
v4:
  - Rebase to latest gvt-staging.
  - Protect vgpu->active access with vgpu_lock.
  - Do not wait gpu idle in vgpu_lock.
v3: update to latest code base
v2: add gvt->lock in function gvt_check_vblank_emulation

Performance comparison on Kabylake platform.
  - Configuration:
    Host: Ubuntu 16.04.
    Guest 1 & 2: Ubuntu 16.04.

glmark2 score comparison:
  - Configuration:
    Host: glxgears.
    Guests: glmark2.
+--------------------------------+-----------------+
| Setup                          | glmark2 score   |
+--------------------------------+-----------------+
| unified lock, iommu=on         | 58~62 (avg. 60) |
+--------------------------------+-----------------+
| unified lock, iommu=igfx_off   | 57~61 (avg. 59) |
+--------------------------------+-----------------+
| per-logic lock, iommu=on       | 60~68 (avg. 64) |
+--------------------------------+-----------------+
| per-logic lock, iommu=igfx_off | 61~67 (avg. 64) |
+--------------------------------+-----------------+

lock_stat comparison:
  - Configuration:
    Stop lock stat immediately after boot up.
    Boot 2 VM Guests.
    Run glmark2 in guests.
    Start perf lock_stat for 20 seconds and stop again.
  - Legend: c - contentions; w - waittime-avg
+------------+-----------------+-----------+---------------+------------+
|            | gvt_lock        |sched_lock | vgpu_lock     | gtt_lock   |
+ lock type; +-----------------+-----------+---------------+------------+
| iommu set  | c     | w       | c  | w    | c    | w      | c   | w    |
+------------+-------+---------+----+------+------+--------+-----+------+
| unified;   | 20697 | 839     |N/A | N/A  | N/A  | N/A    | N/A | N/A  |
| on         |       |         |    |      |      |        |     |      |
+------------+-------+---------+----+------+------+--------+-----+------+
| unified;   | 21838 | 658.15  |N/A | N/A  | N/A  | N/A    | N/A | N/A  |
| igfx_off   |       |         |    |      |      |        |     |      |
+------------+-------+---------+----+------+------+--------+-----+------+
| per-logic; | 1553  | 1599.96 |9458|429.97| 5846 | 274.33 | 0   | 0.00 |
| on         |       |         |    |      |      |        |     |      |
+------------+-------+---------+----+------+------+--------+-----+------+
| per-logic; | 1911  | 1678.32 |8335|445.16| 5451 | 244.80 | 0   | 0.00 |
| igfx_off   |       |         |    |      |      |        |     |      |
+------------+-------+---------+----+------+------+--------+-----+------+

Signed-off-by: Pei Zhang <pei.zhang@intel.com>
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-05-18 12:39:02 +08:00
Changbin Du
e502a2af4c drm/i915/gvt: Provide generic page_track infrastructure for write-protected page
This patch provide generic page_track infrastructure for write-protected
guest page. The old page_track logic gets rewrote and now stays in a new
standalone page_track.c. This page track infrastructure can be both used
by vGUC and GTT shadowing.

The important change is that it uses radix tree instead of hash table.
We don't have a predictable number of pages that will be tracked.

Here is some performance data (duration in us) of looking up a element:
Before: (aka. intel_vgpu_find_tracked_page)
 0.091 0.089 0.090 ... 0.093 0.091 0.087 ... 0.292 0.285 0.292 0.291
After: (aka. intel_vgpu_find_page_track)
 0.104 0.105 0.100 0.102 0.102 0.100 ... 0.101 0.101 0.105 0.105

The hash table has good performance at beginning, but turns bad with
more pages being tracked even no 3D applications are running. As
expected, radix tree has stable duration and very quick.

The overall benchmark (tested with Heaven Benchmark) marginally improved
since this is not the bottleneck. What we benefit more from this change
is scalability.

Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2018-03-06 13:19:20 +08:00
Zhenyu Wang
4fafba2d73 drm/i915/gvt: move write protect handler out of mmio emulation function
It's a bit confusing that page write protect handler is live in
mmio emulation handler. This moves it to stand alone gvt ops.

Also remove unnecessary check of write protected page access
in mmio read handler and cleanup handling of failsafe case.

v2: rebase

Reviewed-by: Xiong Zhang <xiong.y.zhang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-12-22 16:33:50 +08:00
Changbin Du
83164886e4 drm/i915/gvt: Select appropriate mmio list at initialization time
Select appropriate mmio list at initialization time, so we don't need to
do duplicated work at where requires the mmio list.

V2:
  - Add a termination mark of mmio list.

Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-12-08 16:18:19 +08:00
Tina Zhang
e546e281d3 drm/i915/gvt: Dmabuf support for GVT-g
This patch introduces a guest's framebuffer sharing mechanism based on
dma-buf subsystem. With this sharing mechanism, guest's framebuffer can
be shared between guest VM and host.

v17:
- modify VFIO_DEVICE_GET_GFX_DMABUF interface. (Alex)

v16:
- add x_hot and y_hot. (Gerd)
- add flag validation for VFIO_DEVICE_GET_GFX_DMABUF. (Alex)
- rebase 4.14.0-rc6.

v15:
- add VFIO_DEVICE_GET_GFX_DMABUF ABI. (Gerd)
- add intel_vgpu_dmabuf_cleanup() to clean up the vGPU's dmabuf. (Gerd)

v14:
- add PROBE, DMABUF and REGION flags. (Alex)

v12:
- refine the lifecycle of dmabuf.

v9:
- remove dma-buf management. (Alex)
- track the dma-buf create and release in kernel mode. (Gerd) (Daniel)

v8:
- refine the dma-buf ioctl definition.(Alex)
- add a lock to protect the dmabuf list. (Alex)

v7:
- release dma-buf related allocations in dma-buf's associated release
  function. (Alex)
- refine ioctl interface for querying plane info or create dma-buf.
  (Alex)

v6:
- align the dma-buf life cycle with the vfio device. (Alex)
- add the dma-buf related operations in a separate patch. (Gerd)
- i915 related changes. (Chris)

v5:
- fix bug while checking whether the gem obj is gvt's dma-buf when user
  change caching mode or domains. Add a helper function to do it.
  (Xiaoguang)
- add definition for the query plane and create dma-buf. (Xiaoguang)

v4:
- fix bug while checking whether the gem obj is gvt's dma-buf when set
  caching mode or doamins. (Xiaoguang)

v3:
- declare a new flag I915_GEM_OBJECT_IS_GVT_DMABUF in drm_i915_gem_object
  to represent the gem obj for gvt's dma-buf. The tiling mode, caching
  mode and domains can not be changed for this kind of gem object. (Alex)
- change dma-buf related information to be more generic. So other vendor
  can use the same interface. (Alex)

v2:
- create a management fd for dma-buf operations. (Alex)
- alloc gem object's backing storage in gem obj's get_pages() callback.
  (Chris)

Signed-off-by: Tina Zhang <tina.zhang@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-12-04 11:24:33 +08:00
Xiong Zhang
b2d6ef7061 drm/i915/gvt: Let each vgpu has separate opregion memory
Currently every vgpu share a common gvt opregion memory, but
it is freed at vgpu destroy, then the later vgpu doesn't have
opregion memory once the first vgpu is destroyed. This cause
guest function failure like reboot, second or later boot.

This patch allocate and init virt opregion memory for each
vgpu, so this memory could be freed at vgpu destroy.

Signed-off-by: Xiong Zhang <xiong.y.zhang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-11-16 11:48:35 +08:00
Changbin Du
bc7b0be316 drm/i915/gvt: Add basic debugfs infrastructure
We need debugfs entry to expose some debug information of gvt and vGPUs.
The first tool will be added is mmio-diff, which help to find the
difference values of host and vGPU mmio. It's useful for platform
enabling.

This patch just add a basic debugfs infrastructure, each vGPU has its own
sub-folder. Two simple attributes are created as a template.
.
├── num_tracked_mmio
├── vgpu1
|   └── active
└── vgpu2
    └── active

Signed-off-by: Changbin Du <changbin.du@intel.com>
Reviewed-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
2017-11-16 11:48:20 +08:00
fred gao
c5d71cb317 drm/i915/gvt: Move vGPU type related code into gvt file
In this patch, all the vGPU type related code will be merged into
same gvt file and the common interface will be exposed to both
XenGT and KvmGT.

v2:
- remove the useless mdev_* gvt_ops.
  add get_gvt_attr ops for MPT module.
  intel_gvt_{init,cleanup}_vgpu_type_groups are initialized in
  gvt part. (Wang, Zhi)
- set gvt_vgpu_type_groups[i] to NULL. (Zhang,Xiong)

Signed-off-by: fred gao <fred.gao@intel.com>
Reviewed-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
2017-11-16 11:48:19 +08:00
Changbin Du
02d578e5ed drm/i915/gvt: Add support for PCIe extended configuration space
IGD is PCIe device and has extended configuration space. Checking
the binary dump, we can see we have Caps located out of PCI compatible
Configuration Space range.

0x000: 86 80 12 19 17 04 10 00 06 00 00 03 00 00 00 00
0x010: 04 00 00 10 08 00 00 00 0c 00 00 00 08 00 00 00
0x020: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 b9 06
0x030: 00 f8 ff ff 40 00 00 00 00 00 00 00 0b 01 00 00
0x040: 09 70 0c 01 71 26 01 62 c8 00 04 84 00 00 00 00
0x050: c1 00 00 00 39 00 00 00 00 00 00 00 01 00 00 a2
0x060: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
0x070: 10 ac 92 00 00 80 00 10 00 00 00 00 00 00 00 00
0x080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0a0: 00 00 00 00 00 00 00 00 00 00 00 00 05 d0 01 00
0x0b0: 18 00 e0 fe 00 00 00 00 00 00 00 00 00 00 00 00
0x0c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0d0: 01 00 22 00 00 80 00 00 00 00 00 00 00 00 00 00
0x0e0: 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00
0x0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x100: 1b 00 01 20 02 14 00 00 00 00 00 00 00 00 00 00
...

Currently, we only emulate the PCI compatible Configuration Space.
This is okay if we attach vGPU to PCI bus. But when we attach to
a PCI Express bus (when Qemu emulates a Intel Q35 chipset which has
PCIe slot), it will not work. Extended Configuration Space is required
for a PCIe device.

This patch extended the virtual configuration space from 256 bytes
to 4KB bytes. So we are to be a *real* PCIe device. And for the
Extended CapList we keep same to physical GPU.

Cc: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-09-08 14:21:16 +08:00
Ping Gao
c713cb2f9b drm/i915/gvt: Support event based scheduling
This patch decouple the time slice calculation and scheduler, let
other event be able to trigger scheduling without impact the
calculation for QoS.

v2: add only one new enum definition.
v3: fix typo.

Signed-off-by: Ping Gao <ping.a.gao@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-06-08 13:59:16 +08:00
Changbin Du
0e86cc9ccc drm/i915/gvt: implement per-vm mmio switching optimization
Commit ab9da627906a ("drm/i915: make context status notifier head be
per engine") gives us a chance to inspect every single request. Then
we can eliminate unnecessary mmio switching for same vGPU. We only
need mmio switching for different VMs (including host).

This patch introduced a new general API intel_gvt_switch_mmio() to
replace the old intel_gvt_load/restore_render_mmio(). This function
can be further optimized for vGPU to vGPU switching.

To support individual ring switch, we track the owner who occupy
each ring. When another VM or host request a ring we do the mmio
context switching. Otherwise no need to switch the ring.

This optimization is very useful if only one guest has plenty of
workloads and the host is mostly idle. The best case is no mmio
switching will happen.

v2:
  o fix missing ring switch issue. (chuanxiao)
  o support individual ring switch.

Signed-off-by: Changbin Du <changbin.du@intel.com>
Reviewed-by: Chuanxiao Dong <chuanxiao.dong@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-06-08 13:59:15 +08:00
Dave Airlie
856ee92e86 Linux 4.11-rc7
-----BEGIN PGP SIGNATURE-----
 
 iQEcBAABAgAGBQJY881cAAoJEHm+PkMAQRiGG4UH+wa2z6Qet36Uc4nXFZuSMYrO
 ErUWs1QpTDDv4a+LE4fgyMvM3j9XqtpfQLy1n70jfD14IqPBhHe4gytasAf+8lg1
 YvddFx0Yl3sygVu3dDBNigWeVDbfwepW59coN0vI5nrMo+wrei8aVIWcFKOxdMuO
 n72u9vuhrkEnLJuQk7SF+t4OQob9McXE3s7QgyRopmlKhKo7mh8On7K2BRI5uluL
 t0j5kZM0a43EUT5rq9xR8f5pgtyfTMG/FO2MuzZn43MJcZcyfmnOP/cTSIvAKA5U
 1i12lxlokYhURNUe+S6jm8A47TrqSRSJxaQJZRlfGJksZ0LJa8eUaLDCviBQEoE=
 =6QWZ
 -----END PGP SIGNATURE-----

Merge tag 'v4.11-rc7' into drm-next

Backmerge Linux 4.11-rc7 from Linus tree, to fix some
conflicts that were causing problems with the rerere cache
in drm-tip.
2017-04-19 11:07:14 +10:00
Zhi Wang
b79c52aef3 drm/i915/gvt: Activate/de-activate vGPU in mdev ops.
This patch introduces two functions for activating/de-activating vGPU in
mdev ops.

A racing condition was found between virtual vblank emulation and KVGMT
mdev release path. V-blank emulation will emulate and inject V-blank
interrupt for every active vGPU with holding gvt->lock, while in mdev
release path, it will directly release hypervisor handle without changing
vGPU status or taking gvt->lock, so a kernel oops is encountered when
vblank emulation is injecting a interrupt with a invalid hypervisor
handle. (Reported by Terrence)

To solve this problem, we factor out vGPU activation/de-activation from
vGPU creation/destruction path and let KVMGT mdev release ops de-activate
the vGPU before release hypervisor handle. Once a vGPU is de-activated,
GVT-g will not emulate v-blank for it or touch the hypervisor handle.

Fixes: 659643f ("drm/i915/gvt/kvmgt: add vfio/mdev support to KVMGT")
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-03-30 13:37:53 +08:00
Ping Gao
afe04fbe6c drm/i915/gvt: create an idle vGPU
vGPU resource is allocated by scheduler. To account for non-allocated
free cycles, we create an idle vGPU as the placeholder similar to idle task
concept, which is useful to handle some corner cases in scheduling policy.

Signed-off-by: Ping Gao <ping.a.gao@intel.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-03-30 13:34:07 +08:00
Ping Gao
91d0101ad3 drm/i915/gvt: use hrtimer replace delayed_work in scheduler
Currently the scheduler is triggered by delayed_work, which doesn't
provide precision at microsecond level. Move to hrtimer instead for
more accurate control.

Signed-off-by: Ping Gao <ping.a.gao@intel.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-03-30 13:32:24 +08:00
Xu Han
e3476c0021 drm/i915/gvt: Add KBL dispatch logic in each function.
Extend function dispatch logic to support KBL platform.

Signed-off-by: Xu Han <xu.han@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-03-29 15:28:51 +08:00
Chuanxiao Dong
811a4cf273 drm/i915/gvt: enable IOMMU for gvt
gvt driver has a check which doesn't allow to use gvt when host
gpu iommu is enabled. This check can be removed after enable the
iommu support in gvt

Signed-off-by: Chuanxiao Dong <chuanxiao.dong@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-02-09 14:17:15 +08:00
Zhenyu Wang
3de49a7314 drm/i915/gvt: remove detect_host() MPT hook
We only depend on pvinfo register for GVT-g state detection,
not require hypervisor host detect any more.

Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-02-07 17:21:44 +08:00
Zhenyu Wang
d2896e34eb drm/i915/gvt: move intel iommu detection to intel_gvt_init()
Prepare to remove detect_host() hook. Move intel iommu detection early
in intel_gvt_init().

Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-02-07 17:21:35 +08:00
Jike Song
59c0573dfb drm/i915/gvt: init/destroy vgpu_idr properly
An idr should be initialized before use and destroyed afterwards.

Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2017-01-09 11:11:08 +08:00
Jike Song
f30437c5e7 drm/i915/gvt: add KVMGT support
KVMGT is the MPT implementation based on VFIO/KVM. It provides
a kvmgt_mpt ops to gvt for vGPU access mediation, e.g. to
mediate and emulate the MMIO accesses, to inject interrupts
to vGPU user, to intercept the GTT writing and replace it with
DMA-able address, to write-protect guest PPGTT table for
shadowing synchronization, etc. This patch provides the MPT
implementation for GVT, not yet functional due to theabsence
of mdev.

It's built as kvmgt.ko, depends on vfio.ko, kvm.ko and mdev.ko,
and being required by i915.ko. To not introduce hard dependency
in i915.ko, we used indirect symbol reference. But that means
users have to include kvmgt.ko into init ramdisk if their
i915.ko is included.

Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Xiaoguang Chen <xiaoguang.chen@intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-11-10 15:45:39 +08:00
Jike Song
9ec1e66b80 drm/i915/gvt: refactor intel_gvt_io_emulation_ops to be intel_gvt_ops
There are currently 4 methods in intel_gvt_io_emulation_ops
to emulate CFG/MMIO reading/writing for intel vGPU. A possibly
better scope is: add 3 more methods for vgpu create/destroy/reset
respectively, and rename the ops to 'intel_gvt_ops', then pass
it to the MPT module (say the future kvmgt) to use: they are
all methods for external usage.

Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-11-10 15:45:15 +08:00
Jike Song
40df6ea07a drm/i915/gvt: introduce host_init/host_exit to MPT
GVT host needs init/exit hooks to do some initialization/cleanup
work, e.g.: vfio mdev host device register/unregister.

Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-11-10 15:45:13 +08:00
Zhenyu Wang
1f31c82948 drm/i915/gvt: add intel vgpu types support
By providing predefined vGPU types, users can choose which type a vgpu
to create and use, without specifying detailed parameters.

Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
2016-11-10 15:44:54 +08:00
Xiaoguang Chen
b0122f3114 drm/i915/gvt: fix detect_host calling logic
Like other routines, intel_gvt_hypervisor_detect_host returns 0
for success.

Signed-off-by: Xiaoguang Chen <xiaoguang.chen@intel.com>
Signed-off-by: Jike Song <jike.song@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-26 13:23:04 +08:00
Du, Changbin
49129204c3 drm/i915/gvt: get msi cap offset from pdev directly
Linux PCI driver saves the msi and msix capability offset
in pci_dev->msi_cap and pci_dev->msix_cap. We can use msi_cap
in pci_dev directly, no need hardcode.

Signed-off-by: Du, Changbin <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-25 10:37:42 +08:00
Zhenyu Wang
feddf6e866 drm/i915/gvt: clean up intel_gvt.h as interface for i915 core
i915 core should only call functions and structures exposed through
intel_gvt.h. Remove internal gvt.h and i915_pvinfo.h.

Change for internal intel_gvt structure as private handler which
not requires to expose gvt internal structure for i915 core.

v2: Fix per Chris's comment
- carefully handle dev_priv->gvt assignment
- add necessary bracket for macro helper
- forward declartion struct intel_gvt
- keep free operation within same file handling alloc

v3: fix use after free and remove intel_gvt.initialized

v4: change to_gvt() to an inline

Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-20 17:18:30 +08:00
Zhi Wang
be1da7070a drm/i915/gvt: vGPU command scanner
This patch introduces a command scanner to scan guest command buffers.

Signed-off-by: Yulei Zhang <yulei.zhang@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:15:25 +08:00
Zhi Wang
4b63960ebd drm/i915/gvt: vGPU schedule policy framework
This patch introduces a vGPU schedule policy framework, with a timer based
schedule policy module for now

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:15:02 +08:00
Zhi Wang
e473405783 drm/i915/gvt: vGPU workload scheduler
This patch introduces the vGPU workload scheduler routines.

GVT workload scheduler is responsible for picking and executing GVT workload
from current scheduled vGPU. Before the workload is submitted to host i915,
the guest execlist context will be shadowed in the host GVT shadow context.
the instructions in guest ring buffer will be copied into GVT shadow ring
buffer. Then GVT-g workload scheduler will scan the instructions in guest
ring buffer and submit it to host i915.

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:14:50 +08:00
Zhi Wang
04d348ae3f drm/i915/gvt: vGPU display virtualization
This patch introduces the GVT-g display virtualization.

It consists a collection of display MMIO handlers, like power well register
handler, pipe register handler, plane register handler, which will emulate
all display MMIOs behavior to support virtual mode setting sequence for
guest.

Signed-off-by: Bing Niu <bing.niu@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:13:06 +08:00
Zhi Wang
e39c5add32 drm/i915/gvt: vGPU MMIO virtualization
This patch introduces the generic vGPU MMIO emulation intercept
framework.  The MPT modules will request GVT-g core logic to
emulate MMIO read/write through IO emulation operations
callback when hypervisor trapped a guest GTTMMIO read/write.

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:12:59 +08:00
Zhi Wang
4d60c5fd3f drm/i915/gvt: vGPU PCI configuration space virtualization
This patch introduces vGPU PCI configuration space virtualization.

- Adjust the trapped GPFN(Guest Page Frame Number) window of virtual GEN
PCI BAR 0 when guest initializes PCI BAR 0 address.

- Emulate OpRegion when guest touches OpRegion.

- Pass-through a part of aperture to guest when guest initializes
aperture BAR.

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:12:46 +08:00
Zhi Wang
2707e44466 drm/i915/gvt: vGPU graphics memory virtualization
The vGPU graphics memory emulation framework is responsible for graphics
memory table virtualization. Under virtualization environment, a VM will
populate the page table entry with guest page frame number(GPFN/GFN), while
HW needs a page table filled with MFN(Machine frame number). The
relationship between GFN and MFN(Machine frame number) is managed by
hypervisor, while GEN HW doesn't have such knowledge to translate a GFN.

To solve this gap, shadow GGTT/PPGTT page table is introdcued.

For GGTT, the GFN inside the guest GGTT page table entry will be translated
into MFN and written into physical GTT MMIO registers when guest write
virtual GTT MMIO registers.

For PPGTT, a shadow PPGTT page table will be created and write-protected
translated from guest PPGTT page table.  And the shadow page table root
pointers will be written into the shadow context after a guest workload
is shadowed.

vGPU graphics memory emulation framework consists:

- Per-GEN HW platform page table entry bits extract/de-extract routines.
- GTT MMIO register emulation handlers, which will call hypercall to do
GFN->MFN translation when guest write GTT MMIO register
- PPGTT shadow page table routines, e.g. shadow create/destroy/out-of-sync

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:12:33 +08:00
Zhi Wang
c8fe6a6811 drm/i915/gvt: vGPU interrupt virtualization.
This patch introduces vGPU interrupt emulation framework.

The vGPU intrerrupt emulation framework is an event-based interrupt
emulation framework. It's responsible for emulating GEN hardware interrupts
during emulating other HW behaviour.

It consists several components:

- Descriptions of interrupt register bit
- Upper level <-> lower level interrupt mapping
- GEN HW IER/IMR/IIR register emulation routines
- Event-based interrupt propagation interface

When a GVT-g component wants to inject an interrupt to a VM during a
emulation, first it should specify the event needs to be emulated and the
framework will deal with the rest of emulation:

- Generating related virtual IIR bit according to virtual IER and IMRs,
- Generate related virtual upper level virtual IIR bit accodring to the
per-platform interrupt mapping
- Injecting a MSI to VM

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:12:22 +08:00
Zhi Wang
579cea5f30 drm/i915/gvt: golden virtual HW state management
Each vGPU expects a golden virtual HW state, which is just the state after
system is freshly powered on. GVT-g will try to load the golden virtual HW
state via kernel firmware interface.

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:11:46 +08:00
Zhi Wang
12d14cc43b drm/i915/gvt: Introduce a framework for tracking HW registers.
This patch introduces a framework for tracking HW registers on different
GEN platforms.

Accesses to GEN HW registers from VMs will be trapped by hypervisor. It
will forward these emulation requests to GVT-g device model, which
requires this framework to search for related register descriptions.

Each MMIO entry in this framework describes a GEN HW registers, e.g.
offset, length, whether it contains RO bits, whether it can be accessed by
LRIs...and also emulation handlers for emulating register reading and
writing.

- Use i915 MMIO register definition & statement.(Joonas)

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:11:33 +08:00
Zhi Wang
28a60dee2c drm/i915/gvt: vGPU HW resource management
This patch introduces the GVT-g vGPU HW resource management. Under
GVT-g virtualizaion environment, each vGPU requires portions HW
resources, including aperture, hidden GM space, and fence registers.

When creating a vGPU, GVT-g will request these HW resources from host,
and return them to host after a vGPU is destroyed.

Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>
2016-10-14 18:11:19 +08:00
Zhi Wang
0ad35fed61 drm/i915: gvt: Introduce the basic architecture of GVT-g
This patch introduces the very basic framework of GVT-g device model,
includes basic prototypes, definitions, initialization.

v12:
- Call intel_gvt_init() in driver early initialization stage. (Chris)

v8:
- Remove the GVT idr and mutex in intel_gvt_host. (Joonas)

v7:
- Refine the URL link in Kconfig. (Joonas)
- Refine the introduction of GVT-g host support in Kconfig. (Joonas)
- Remove the macro GVT_ALIGN(), use round_down() instead. (Joonas)
- Make "struct intel_gvt" a data member in struct drm_i915_private.(Joonas)
	- Remove {alloc, free}_gvt_device()
	- Rename intel_gvt_{create, destroy}_gvt_device()
	- Expost intel_gvt_init_host()
- Remove the dummy "struct intel_gvt" declaration in intel_gvt.h (Joonas)

v6:
- Refine introduction in Kconfig. (Chris)
- The exposed API functions will take struct intel_gvt * instead of
void *. (Chris/Tvrtko)
- Remove most memebers of strct intel_gvt_device_info. Will add them
in the device model patches.(Chris)
- Remove gvt_info() and gvt_err() in debug.h. (Chris)
- Move GVT kernel parameter into i915_params. (Chris)
- Remove include/drm/i915_gvt.h, as GVT-g will be built within i915.
- Remove the redundant struct i915_gvt *, as the functions in i915
will directly take struct intel_gvt *.
- Add more comments for reviewer.

v5:
Take Tvrtko's comments:
- Fix the misspelled words in Kconfig
- Let functions take drm_i915_private * instead of struct drm_device *
- Remove redundant prints/local varible initialization

v3:
Take Joonas' comments:
- Change file name i915_gvt.* to intel_gvt.*
- Move GVT kernel parameter into intel_gvt.c
- Remove redundant debug macros
- Change error handling style
- Add introductions for some stub functions
- Introduce drm/i915_gvt.h.

Take Kevin's comments:
- Move GVT-g host/guest check into intel_vgt_balloon in i915_gem_gtt.c

v2:
- Introduce i915_gvt.c.
It's necessary to introduce the stubs between i915 driver and GVT-g host,
as GVT-g components is configurable in kernel config. When disabled, the
stubs here do nothing.

Take Joonas' comments:
- Replace boolean return value with int.
- Replace customized info/warn/debug macros with DRM macros.
- Document all non-static functions like i915.
- Remove empty and unused functions.
- Replace magic number with marcos.
- Set GVT-g in kernel config to "n" by default.

Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Zhi Wang <zhi.a.wang@intel.com>
Link: http://patchwork.freedesktop.org/patch/msgid/1466078825-6662-5-git-send-email-zhi.a.wang@intel.com
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2016-06-17 20:35:06 +01:00