Commit Graph

183 Commits

Author SHA1 Message Date
Oded Gabbay
b8cbab042c drm/amdkfd: Allow user to limit only queues per device
This patch replaces the two current amdkfd module parameters with a new one.

The current parameters that are being replaced are:

- Maximum number of HSA processes
- Maximum number of queues per process

The new parameter that replaces them is called "Maximum queues per device"

This replacement achieves two goals:

- Allows the user to have as many HSA processes as it wants (until
  a maximum of 512 HSA processes in Kaveri).

- Removes the limitation the user had on maximum number of queues per HSA
  process. E.g. the user can now have processes which only have one queue and
  other processes which have hundreds of queues, while before the user
  couldn't have more than 128 queues per process (as default).

The default value of the new parameter is 4096 (32 * 128, which were the
defaults of the old parameters). There is almost no additional GART memory
required for the default case. As a reminder, this amount of queues requires a
little bit below 4MB of GART memory.

v2:
In addition, This patch defines a new counter for queues accounting in the DQM
structure. This is done because the current counter only counts active queues
which allows the user to create more queues than the
max_num_of_queues_per_device module parameter allows.

However, we need the current counter for the runlist packet build process, so
the solution is to have a dedicated counter for this accounting.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Ben Goz <ben.goz@amd.com>
2015-01-18 13:18:01 +02:00
Ben Goz
cb2ac44128 drm/amdkfd: Fix description of sched_policy module parameter
Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-18 13:18:01 +02:00
Ben Goz
f046bfdf73 drm/amdkfd: PQM handle queue creation fault
If the first queue created was failed on DQM then PQM should
unregister the process from DQM.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-15 17:14:47 +02:00
Oded Gabbay
939f4a20a7 drm/amdkfd: Remove sync_with_hw() from amdkfd
This patch completely removes the sync_with_hw() because it was broken and
actually there is no point of using it.

This function was used to:

- Make sure that the submitted packet to the HIQ (which is a kernel queue) was
  read by the CP. However, it was discovered that the method this function used
  to do that (checking wptr == rptr) is not consistent with how the actual CP
  firmware works in all cases.

- Make sure that the queue is empty before issuing the next packet. To achieve
  that, the function blocked amdkfd from continuing until the recently
  submitted packet was consumed. However, the acquire_packet_buffer() already
  checks if there is enough room for a new packet so calling sync_with_hw() is
  redundant.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-15 12:07:48 +02:00
Oded Gabbay
c51841fbbb drm/amdkfd: Remove unused function busy_wait()
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-15 12:04:10 +02:00
Oded Gabbay
99331a51cc drm/amdkfd: Replace cpu_relax() with schedule() in DQM
In order not to occupy the current core and thus prevent the core from
servicing IOMMU PPR requests, this patch replaces the call in DQM to
cpu_relax() with a call to schedule().

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-15 12:01:10 +02:00
Ben Goz
f0ec5b9905 drm/amdkfd: Fix for-loop when allocating HQD (non-HWS)
This patch fixes a minor bug in allocate_hqd(), where the loop run from the
next-to-allocate pipe until the number of pipes.

This is wrong because we need to consider the possibility where
next-to-allocate pipe is not 0, and thus, the for-loop only checks part of the
pipes and doesn't wrap-around, as it supposed to do.

Therefore, we add another counting variable to make sure we go over all the
pipes, regardless of where we start to look at the first iteration of the loop.

This bug only affected non-HWS mode. In HWS mode, the CP fw is responsible for
allocating the HQD.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-13 11:18:06 +02:00
Oded Gabbay
8dfe58b206 drm/amdkfd: Fix sparse warning (different address space)
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-08 16:46:16 +02:00
Michel Dänzer
6ee0ad2a7f drm/amdkfd: Drop interrupt SW ring buffer
The work queue couldn't reliably prevent the SW ring buffer from
overflowing, so dmesg was spammed by

 kfd kfd: Interrupt ring overflow, dropping interrupt.

messages when running e.g. the Atlantis Substance demo from
https://wiki.unrealengine.com/Linux_Demos on Kaveri.

Since the SW ring buffer doesn't actually do anything at this point, just
remove it for now. When actual interrupt processing code is added to
amdkfd, it should try to do things immediately and only defer to work
queues when necessary.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-08 13:27:15 +09:00
Oded Gabbay
76baee6c73 drm/amdkfd: rewrite kfd_ioctl() according to drm_ioctl()
This patch changes kfd_ioctl() to be very similar to drm_ioctl().

The patch defines an array of amdkfd_ioctls, which maps IOCTL definition to the
ioctl function.

The kfd_ioctl() uses that mapping to call the appropriate ioctl function,
through a function pointer.

This patch also declares a new typedef for the ioctl function pointer.

v2: Renamed KFD_COMMAND_(START|END) to AMDKFD_...

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
2015-01-06 19:44:36 +02:00
Oded Gabbay
b81c55db10 drm/amdkfd: reformat IOCTL definitions to drm-style
This patch reformats the ioctl definitions in kfd_ioctl.h to be similar to the
drm ioctls definition style.

v2: Renamed KFD_COMMAND_(START|END) to AMDKFD_...

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
2015-01-06 19:44:36 +02:00
Oded Gabbay
524a640444 drm/amdkfd: Do copy_to/from_user in general kfd_ioctl()
This patch moves the copy_to_user() and copy_from_user() calls from the
different ioctl functions in amdkfd to the general kfd_ioctl() function, as
this is a common code for all ioctls.

This was done according to example taken from drm_ioctl.c

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
2015-01-06 19:44:26 +02:00
Ben Goz
2030664b70 drm/amdkfd: unmap VMID<-->PASID when relesing VMID (non-HWS)
This patch fixes a bug where deallocate_vmid() didn't actually unmap the
VMID<-->PASID mapping (in the registers).
That can cause undefined behavior.

This bug only occurs in non-HWS mode.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-05 15:48:28 +02:00
Ben Goz
030e416b4f drm/amdkfd: Load mqd to hqd in non-HWS mode
This patch fixes a bug in DQM, where the MQD of a newly created compute queue
is not loaded to an HQD slot. As a result, the CP never reads packets from this
queue.

This bug happens only in non-HWS (hardware scheduling) mode. In HWS mode, the
CP is responsible of loading MQDs to HQDs slots.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 21:46:44 +02:00
Ben Goz
b64b8afcca drm/amd: Fixing typos in kfd<->kgd interface
Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-09 12:00:09 +02:00
Sasha Levin
68d0cb49f8 amdkfd: actually allocate longs for the pasid bitmask
Commit "amdkfd: use sizeof(long) granularity for the pasid bitmask" calculated
the number of longs it will need, but ended up allocating that number of
bytes rather than longs.

Fix that silly error and allocate the amount of data really required.

Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-28 11:44:37 -05:00
Oded Gabbay
0cb989c0c6 amdkfd: Remove duplicate include
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-05 10:40:34 +02:00
Ben Goz
8dfead6c28 amdkfd: Fixing topology bug in building sysfs nodes
Original code sent always 0 as the index number of the node. This patch fixes
this bug by sending a variable which is incremented per node.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Reviewed-by: Oded Gabbay <oded.gabbay@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-02 16:41:08 +02:00
Oded Gabbay
b6ffbab813 amdkfd: Fix accounting of device queues
This patch fixes a device QCM bug, where the number of queues were not
counted correctly for the operation of update queue. The count was incorrect
as there was no regard to the previous state of the queue.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-07 22:27:24 +02:00
Ben Goz
6898f0a568 drm/amdkfd: Add initial VI support for KQ
This patch starts to add support for the VI APU in the KQ (kernel queue)
module.

Because most (more than 90%) of the KQ code is shared among AMD's APUs, we
chose a design that performs most/all the code in the shared KQ file
(kfd_kernel_queue.c). If there is H/W specific code to be executed,
than it is written in an asic-specific extension function for that H/W.

That asic-specific extension function is called from the shared function at the
appropriate time. This requires that for every asic-specific extension function
that is implemented in a specific ASIC, there will be an equivalent
implementation in ALL ASICs, even if those implementations are just stubs.

That way we achieve:

- Maintainability: by having one copy of most of the code, we only need to
  fix bugs at one locations

- Readability: very clear what is the shared code and what is done per ASIC

- Extensibility: very easy to add new H/W specific files/functions

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-12-02 16:38:57 +02:00
Oded Gabbay
443fbd5f11 drm/amdkfd: Encapsulate KQ functions in ops structure
This patch does some re-org on the kernel_queue structure. It takes out
all the function pointers from the structure and puts them in a new structure,
called kernel_queue_ops. Then, it puts an instance of that structure
inside kernel_queue.

This re-org is done to prepare the KQ module to support more than one AMD APU
(Kaveri).

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-12 15:53:44 +02:00
Ben Goz
a22fc85495 drm/amdkfd: Add initial VI support for DQM
This patch starts to add support for the VI APU in the DQM module.

Because most (more than 90%) of the DQM code is shared among AMD's APUs, we
chose a design that performs most/all the code in the shared DQM file
(kfd_device_queue_manager.c). If there is H/W specific code to be executed,
than it is written in an asic-specific extension function for that H/W.

That asic-specific extension function is called from the shared function at the
appropriate time. This requires that for every asic-specific extension function
that is implemented in a specific ASIC, there will be an equivalent
implementation in ALL ASICs, even if those implementations are just stubs.

That way we achieve:

- Maintainability: by having one copy of most of the code, we only need to
  fix bugs at one locations

- Readability: very clear what is the shared code and what is done per ASIC

- Extensibility: very easy to add new H/W specific files/functions

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-12 14:28:46 +02:00
Oded Gabbay
45c9a5e429 drm/amdkfd: Encapsulate DQM functions in ops structure
This patch does some re-org on the device_queue_manager structure. It takes out
all the function pointers from the structure and puts them in a new structure,
called device_queue_manager_ops. Then, it puts an instance of that structure
inside device_queue_manager.

This re-org is done to prepare the DQM module to support more than one AMD APU
(Kaveri).

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-12 14:26:10 +02:00
Oded Gabbay
9216ed2940 drm/amdkfd: Don't BUG on freeing GART sub-allocation
Instead of creating a BUG if trying to free a NULL GART sub-allocation object,
just return 0 (success).

This is done to mirror behavior of kfree.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-12 22:34:21 +02:00
Alexey Skidanov
dd59239a98 amdkfd: init aperture once per process
Since the user space may call open() more that once from the same process,
the aperture initialization should be moved from kfd_open()

Signed-off-by: Alexey Skidanov <Alexey.Skidanov@amd.com>
Reviewed-by: Oded Gabbay <oded.gabbay@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-18 13:56:23 +02:00
Oded Gabbay
f1386fbc2b amdkfd: Display MEC fw version in topology node
This patch displays the firmware version of the microcode that is currently
running in the MEC.
This is needed for the HSA RT, so it could differentiate its behavior based on
fw version. e.g. workarounds for bugs in fw

v2: Send the KGD_ENGINE_MEC1 as a parameter to the get_fw_version()

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-11-09 12:46:56 +02:00
Oded Gabbay
a18069c132 amdkfd: Disable support for 32-bit user processes
This patch checks if the process that opens the /dev/kfd device is 32-bit
process. If so, it returns -EPERM and prints a warning message in dmesg.

This is done to prevent 32-bit user processes from using amdkfd, and hence, HSA
features.

AMD's HSA userspace stack will also support only 64-bit processes on Linux.

Reviewed-by: Alexey Skidanov <alexey.skidanov@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-05 22:01:35 +02:00
Oded Gabbay
a550bb3d53 amdkfd: Set *buffer_ptr to NULL in case of error
In function acquire_packet_buffer() we may return -ENOMEM. In that case, we
should set the *buffer_ptr to NULL, so that calling functions which check the
*buffer_ptr value as a criteria for success, will know that
acquire_packet_buffer() failed.

Reviewed-by: Alexey Skidanov <alexey.skidanov@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-04 14:09:02 +02:00
Sasha Levin
c448a142a7 amdkfd: use atomic allocations within srcu callbacks
srcu callbacks are running in atomic context, we can't allocate using
__GFP_WAIT.

Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-03 10:19:36 -05:00
Sasha Levin
aeda036c37 amdkfd: use sizeof(long) granularity for the pasid bitmask
All the bit operations (such as find_first_zero_bit()) read sizeof(long) bytes
at a time. If we allocated less than sizeof(long) bytes for the bitmask we
would be accessing invalid memory when working with the bitmask.

Change the allocator to allocate sizeof(long) multiples for the bitmask.

Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-12-03 09:26:25 -05:00
Dan Carpenter
9cf4a28131 amdkfd: delete some dead code
This is dead code.  We don't need to unbind here, we can just return
directly.

Reviewed-by: Oded Gabbay <oded.gabbay@amd.com>
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-25 19:43:29 +03:00
Oded Gabbay
6f9d54fd6e amdkfd: Fix memory leak of mqds on dqm fini
The mqds array members are not freed when dqm is uninitialized.

Reviewed-by: Ben Goz <Ben.Goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-25 15:16:38 +02:00
Dan Carpenter
e048a0b260 amdkfd: fix an error handling bug in pqm_create_queue()
The call to kernel_queue_uninit(NULL) will trigger a BUG(), and also the
error code is incorrect.

Fixes: 45102048f7 ('amdkfd: Add process queue manager module')
Reviewed-by: Oded Gabbay <oded.gabbay@amd.com>
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-25 13:24:51 +03:00
Dan Carpenter
66333cb3d7 amdkfd: fix some error handling in ioctl
There is a typo here so the errors from kfd_bind_process_to_device()
are not detected.

Reviewed-by: Oded Gabbay <oded.gabbay@amd.com>
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-25 13:21:30 +03:00
Oded Gabbay
a49493b548 amdkfd: Remove DRM_AMDGPU dependency from Kconfig
This patch removes the dependency of amdkfd upon DRM_AMDGPU symbol in amdkfd's
Kconfig file.

This is done because amdgpu driver is not yet upstreamed and therefore,
DRM_AMDGPU symbol is not present in any Kconfig file.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-21 22:36:09 +02:00
Oded Gabbay
824cb7d136 amdkfd: explicitely include io.h in kfd_doorbell.c
This patch fixes a compilation error when using certain configuration by
including the file io.h in kfd_doorbell.c

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-21 22:04:44 +02:00
Oded Gabbay
abc9d3e3b9 amdkfd: Clear ctx cb before suspend
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-09 22:36:22 +02:00
Alexey Skidanov
52a5fdce13 amdkfd: Instead of using get function, use container_of
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alexey Skidanov <Alexey.Skidanov@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-19 17:07:00 +02:00
Oded Gabbay
9a5634a729 amdkfd: use schedule() in sync_with_hw
amdkfd uses cpu_relax() in its sync_with_hw() function. Because cpu_relax() is
defined as 'REP; NOP' on x86_64, it will block the CPU from servicing
IOMMU PPR requests.

This may cause a deadlock, because sync_with_hw() won't be completed
until the PPR request has been served.

Therefore, we need to use schedule() instead of cpu_relax() as it is the
minimum requirement to allow other threads to execute.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-17 13:18:32 +02:00
Jay Cornwall
f5d896bbd0 amdkfd: Fix memory leak on process deregistration
struct device_process_node was allocated during process registration but
not released at process deregistration.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 11:52:16 -06:00
Oded Gabbay
5cd78de526 amdkfd: add __iomem attribute to doorbell_ptr
This patch was done due to sparse warning. It changes the definition of
doorbell_ptr in queue_properties to be with __iomem attribute, so it would
match the type which the doorbell module functions are returning.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 16:14:56 +02:00
Oded Gabbay
d80d19bd50 amdkfd: fence_wait_timeout() can be static
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 15:54:05 +02:00
Oded Gabbay
20981e6801 amdkfd: is_occupied() can be static
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 15:50:53 +02:00
Oded Gabbay
585dbf3842 amdkfd: Fix sparse warnings in kfd_flat_memory.c
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 15:49:49 +02:00
kbuild test robot
7347a6cbf1 amdkfd: pqm_get_kernel_queue() can be static
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 17:16:23 +08:00
kbuild test robot
5ef360eab7 amdkfd: test_kq() can be static
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 16:08:14 +08:00
Oded Gabbay
16b9201c62 amdkfd: Fix sparse warnings in kfd_topology.c
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 15:41:33 +02:00
Oded Gabbay
4307d8f6e5 amdkfd: Fix sparse warnings in kfd_chardev.c
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-20 15:37:13 +02:00
Oded Gabbay
ecd5c9821c amdkfd: Implement the Get Version IOCTL
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-11-02 12:18:29 +02:00
Ben Goz
c2e1b3a496 drm/amdkfd: Fix logic of destroy_queue_nocpsch()
This patch rewrites destroy_queue_nocpsch() as the current logic that is
implemented in the function is completely flawed.

This function is used only in non-HWS mode.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
2014-08-18 14:55:59 +03:00
Ben Goz
4b8f589b05 drm/amdkfd: Change MQD manager to be H/W specific
The MQDs for CI and VI are different. Therefore, the MQD manager module need to
be H/W specific.

This patch splits the current MQD manager into three files:

- kfd_mqd_manager.c, which contains common functions and initializes the
  specific mqd manager module according to the H/W

- kfd_mqd_manager_cik.c, which contains Kaveri specific functions. This is
  basically the old kfd_mqd_manager.c

- kfd_mqd_manager_vi.c, which will contain VI specific functions. Currently it
  is not implemented except for returning NULL on initialization.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 11:24:25 +02:00
Ben Goz
0da7558c69 drm/amdkfd: Add asic property to kfd_device_info
This patch adds a new property to kfd_device_info structure. That structure
holds information that is H/W specific.

The new property is called asic_family and its purpose is to distinguish
between different asic families in amdkfd operations, mainly in QCM (queue
control & management)

This patch also adds a new enum, to select different ASICs. We set the current
kfd_device_info instance as Kaveri and create a new instance which describes
the new AMD APU, codenamed 'Carrizo'.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-01 17:10:01 +02:00
Ben Goz
85d258f9a7 drm/amdkfd: Make KFD_MQD_TYPE enum types H/W agnostic
As the MQD types are common across all AMD GPUs/APUs, let's remove the CIK part
from the name.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 10:36:30 +02:00
Ben Goz
ff3d04a171 drm/amdkfd: Add new VI-specific queue properties
This patch adds new fields to the queue_properties structure. The new fields
are relevant only for queues running on AMD GPU VI architecture.

The eop_ring_buffer_address and eop_ring_buffer_size describe an
end-of-pipe queue which is assigned to the MQD. In CI, the EOP queue was per
pipeline and in VI it is per queue.

The ctx_save_restore_area_address and ctx_save_restore_area_size describe a
memory area that is designated to allow the CP to do context save/restore in
mid-wave state.

This patch also modifies the set_queue_properties_from_user() (called from
kfd_ioctl_create_queue()) to check and copy those new parameters.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-04 10:37:18 +02:00
Oded Gabbay
71273adc52 drm/amdkfd: Don't include header files from radeon
Because amdkfd will need to work both with radeon and amdgpu, don't include
header files that are in radeon's folder.

Instead, use the common amd include folder and move amdkfd specific defines to
amdkfd header files.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-02 23:18:54 +02:00
Ben Goz
fe50280420 drm/amdkfd: Remove call to deprecated init_memory interface
This patch removes a call to kfd-->kgd interface function that is doing H/W
initialization. That function is moved into radeon to be part of the common
H/W initialization sequence. The interface function will be deleted.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2014-10-26 18:07:34 +02:00
Oded Gabbay
a86aa3ca5a drm/amdkfd: Using new gtt sa in amdkfd
This patch change the calls throughout the amdkfd driver from the old kfd-->kgd
interface to the new kfd gtt sa inside amdkfd

v2: change the new call in sdma code that appeared because of the sdma feature

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:10 +02:00
Oded Gabbay
73a1da0bb3 drm/amdkfd: Allocate gart memory using new interface
This patch changes the calls to allocate the gart memory for amdkfd from the
old interface (radeon_sa) to the new one (kfd_gtt_sa)

The new gart sub-allocator is initialized with chunk size equal to 512 bytes.
This is because the KV MQD is 512 Bytes and most of the sub-allocations are
MQDs.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:09 +02:00
Oded Gabbay
e18e794e6b drm/amdkfd: Fixed calculation of gart buffer size
This patch makes the gart's buffer size calculation more accurate. This buffer
is needed per GPU.

It takes into account maximum number of MQDs, runlist packets, kernel queues
and reserves 512KB for other misc allocations.

The total size is just shy of 4MB, for 32 processes and 128 queues per
process, which are the defaults for amdkfd kernel module parameters.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:09 +02:00
Oded Gabbay
6e81090b2e drm/amdkfd: Add kfd gtt sub-allocator functions
This patch adds new kfd gtt sub-allocator functions that service the amdkfd
driver when it wants to use gtt memory.

The sub-allocator uses a bitmap to handle the memory area that was transferred
to it during init. It divides the memory area into chunks, according to chunk
size parameter.

The allocation function will allocate contiguous chunks from that memory area,
according to the requested size. If the requested size is smaller than the
chunk size, a single chunk will be allocated.

v2: Do some more verifications on parameters that are passed into
kfd_gtt_sa_init()

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:08 +02:00
Oded Gabbay
36b5c08f09 drm/amdkfd: Add gtt sa related data to kfd_dev struct
This patch adds new fields to kfd_dev struct that are necessary for the new kfd
gtt sa module

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alexey Skidanov <Alexey.skidanov@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:08 +02:00
Ben Goz
85dfaef341 drm/amdkfd: Pass queue type to pqm_create_queue()
This patch passes the correct queue type to pqm_create_queue() instead of a
fixed KFD_QUEUE_TYPE_COMPUTE type.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:06 +02:00
Ben Goz
3385f9dd64 drm/amdkfd: Identify SDMA queue in create queue ioctl
This patch adds a check to the create queue ioctl path, which identifies SDMA
queue type that is sent by userspace.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:05 +02:00
Ben Goz
bcea308175 drm/amdkfd: Add SDMA user-mode queues support to QCM
This patch adds support for SDMA user-mode queues to the QCM - the Queue
management system that manages queues-per-device and queues-per-process.

v2: Remove calls to interface function that initializes sdma engines.

v3: Use the new names of some of the defines.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:05 +02:00
Ben Goz
77669eb87a drm/amdkfd: Add SDMA mqd support
This patch adds support for SDMA mqd operations:
- init_mqd_sdma
- uninit_mqd_sdma
- load_mqd_sdma
- update_mqd_sdma
- destroy_mqd_sdma
- is_occupied_sdma

It also adds SDMA queue information to some private structures of amdkfd.

v3: Use the new names of some of the defines.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
2015-01-09 22:26:04 +02:00
Alexey Skidanov
093c7d8cfd drm/amdkfd: Process-device data creation and lookup split
This patch splits the current kfd_get_process_device_data() to two
functions, one that specifically creates a pdd and another one which
just do lookup.

This is done to enhance the readability and maintainability of the code.

Signed-off-by: Alexey Skidanov <Alexey.Skidanov@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-09 22:25:58 +02:00
Alexey Skidanov
f7c826ad38 drm/amdkfd: Add number of watch points to topology
This patch adds the number of watch points to the node capabilities in the
topology module

Signed-off-by: Alexey Skidanov <Alexey.Skidanov@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2015-01-09 22:25:55 +02:00
Alexey Skidanov
775921edc1 amdkfd: Implement the Get Process Aperture IOCTL
v3: Fixed debug messages

Signed-off-by: Alexey Skidanov <Alexey.Skidanov@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 01:49:36 +03:00
Evgeny Pinchuk
4fac47c820 amdkfd: Implement the Get Clock Counters IOCTL
Signed-off-by: Evgeny Pinchuk <evgeny.pinchuk@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 01:47:58 +03:00
Andrew Lewycky
41a286fa54 amdkfd: Implement the Set Memory Policy IOCTL
Signed-off-by: Andrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 01:46:17 +03:00
Oded Gabbay
39b027d957 amdkfd: Implement the create/destroy/update queue IOCTLs
v3: Removed the use of internal typedefs, fixed debug prints, added checks
    for parameters and moved to using doorbell address from user

v4: Extracted some of the code in the create queue ioctl to a different
    function that may be also called from other ioctls in the future.
    Also fixed the check of the ring size argument.

v5:

Add support for AQL queues creation to enable working with open-source HSA
runtime

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-10-19 23:46:40 +03:00
Andrew Lewycky
b3f5e6b441 amdkfd: Add interrupt handling module
This patch adds the interrupt handling module, in kfd_interrupt.c, and its
related members in different data structures to the amdkfd driver.

The amdkfd interrupt module maintains an internal interrupt ring per amdkfd
device. The internal interrupt ring contains interrupts that needs further
handling. The extra handling is deferred to a later time through a workqueue.

There's no acknowledgment for the interrupts we use. The hardware simply queues
a new interrupt each time without waiting.

The fixed-size internal queue means that it's possible for us to lose
interrupts because we have no back-pressure to the hardware.

v3:

Move amdkfd from drm/radeon/ to drm/amd/
Change device init
Made sure spin lock is taken only if init is complete
Moved bool field to the end of the structure

Signed-off-by: Andrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 01:37:30 +03:00
Ben Goz
64c7f8cf79 amdkfd: Add device queue manager module
The queue scheduler divides into two sections, one section is process bounded
and the other section is device bounded.
The device bounded section is handled by this module.
The DQM module handles queue setup, update and tear-down from the device side.
It also supports suspend/resume operation.

v3: Changed device_init, added the use of the new gart allocation functions an
Added documentation.

v4:

Fixed a race in DQM queue scheduler where dqm->lock must be held when accessing
dqm->queue_count and dqm->processes_count. This fixes runlist IB allocation
failures when DQM is under load.

Fixed race in DQM queue destruction where queues being destroyed must be
removed from qpd->queues_list prior to preemption, or concurrent queue
creation activity may reschedule them while their MQD is destroyed.

Fixed EOP queue size setting in CP_HPD_EOP_CONTROL, because the size is
specified as (log2(size_dwords)-1). The previous calculation assumed the
size was specified in bytes, which caused interference between EOP queues
when multiple MEC pipelines were active.

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Change format of mqd structure to match latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime
Remove unused unmap_queue function
Various fixes (Style, typos)

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Jay Cornwall <jay.cornwall@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 01:27:00 +03:00
Ben Goz
45102048f7 amdkfd: Add process queue manager module
The queue scheduler divides into two sections, one section is process bounded
and the other section is device bounded.
The process bounded section is handled by this module. The PQM handles usermode
queue setup, updates and tear-down.

v3:

Used kernel parameter to limit queues per process instead of define
Added use of doorbell address from user

v4:

Modified pqm_create_queue so that only when creating usermode queues the
driver should return the queue properties to the userspace.

Added an info message print when no more queues can be opened because of the
queue per process limitation

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Various fixes

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 01:04:10 +03:00
Ben Goz
241f24f823 amdkfd: Add packet manager module
The packet manager module builds PM4 packets for the sole use of the CP
scheduler. Those packets are used by the HIQ to submit runlists to the CP.

v3:

Removed include of cik_mqds.h
Changed lower_32/upper_32 calls to use linux macros
Used new gart allocation functions
Added documentation

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Change format of mqd structure to match latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime
Always chain runlist if you have more than 1 process or if you have
over-subscription over the number of queues.
Various fixes (typos, style)

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 00:55:28 +03:00
Ben Goz
31c21fece7 amdkfd: Add module parameter of scheduling policy
This patch adds a new parameter to the amdkfd driver. This parameter enables
the user to select the scheduling policy of the CP. The choices are:

* CP Scheduling with support for over-subscription
* CP Scheduling without support for over-subscription
* Without CP Scheduling

Note that the third option (Without CP scheduling) is only for debug purposes
and bringup of new H/W. As such, it is _not_ guaranteed to work at all times on
all H/W versions.

v3: Fixed description of parameter, changed the permissions to read_only, added
a verification of the value and added documentation

v5: Set default sched_policy to HWS as it is now supported by firmware

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 00:48:28 +03:00
Ben Goz
ed6e6a3487 amdkfd: Add kernel queue module
The kernel queue module enables the amdkfd to establish kernel queues, not
exposed to user space.

The kernel queues are used for HIQ (HSA Interface Queue) and DIQ (Debug
Interface Queue) operations

v3: Removed use of internal typedefs and added use of the new gart allocation
functions

v4: Fixed a miscalculation in kernel queue wrapping

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Change format of mqd structure to match latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime
Add define for kernel queue size
Various fixes

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 00:45:35 +03:00
Ben Goz
6e99df5741 amdkfd: Add mqd_manager module
The mqd_manager module handles MQD data structures.
MQD stands for Memory Queue Descriptor, which is used by the H/W to
keep the usermode queue state in memory.

v3:

Removed new typedefs
Removed pragma pack 4
Remove cik_mqds.h file
Changed lower_32/upper_32 calls to use linux macros
Used new gart allocation functions
Added documentation

v4:

Added missing initialization of the addr field in init_mqd()

Setting the hqd persistent.preload_req bit ON so that when queues switches
on/off, their context will kept and read from the mqd when the cp reassign
them, and thus the dispatched workload context kept consistent without any
interrupts.

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Change format of mqd structure to match latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime.
Various fixes

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 00:36:17 +03:00
Ben Goz
ed8aab4594 amdkfd: Add queue module
The queue module enables allocating and initializing queues uniformly.

v3: Removed typedef and redundant memset call. Broke long pr_debug print to one
liners and Added documentation.

v5: Move amdkfd from drm/radeon/ to drm/amd/

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 00:18:51 +03:00
Oded Gabbay
b17f068a09 amdkfd: Add binding/unbinding calls to amd_iommu driver
This patch adds the functions to bind and unbind pasid
from a device through the amd_iommu driver.

The unbind function is called when the mm_struct of the
process is released.

The bind function is not called here because it is called
only in the IOCTLs which are not yet implemented at this
stage of the patchset.

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-17 00:06:27 +03:00
Oded Gabbay
19f6d2a660 amdkfd: Add basic modules to amdkfd
This patch adds the process module and three helper modules:

- kfd_process, which handles process which open /dev/kfd

- kfd_doorbell, which provides helper functions for doorbell allocation,
  release and mapping to userspace

- kfd_pasid, which provides helper functions for pasid allocation and release

- kfd_aperture, which provides helper functions for managing the LDS, Local GPU
  memory and Scratch memory apertures of the process

This patch only contains the basic kfd_process module, which doesn't contain
the reference to the queue scheduler. This was done to allow easier code review.

Also, this patch doesn't contain the calls to the IOMMU driver for binding the
pasid to the device. Again, this was done to allow easier code review

The kfd_process object is created when a process opens /dev/kfd and is closed
when the mm_struct of that process is teared-down.

v3:

Removed kfd_vidmem.c file
Replaced direct mmput call to mmu_notifier release
Removed typedefs
Moved bool field to end of the structure
Added new kernel params for gart usage limitation
Added initialization of sa manager
Fixed debug messages
Remove support for LDS in 32 bit
Changed code to support mmap of doorbell pages from userspace
Added documentation for apertures

v4: Replaced RCU by SRCU for kfd_process list management

v5:

Move amdkfd from drm/radeon/ to drm/amd/
Rename kfd_aperture.c to kfd_flat_memory.c
Protect against multiple init calls
MQD size is H/W dependent so moved it to device info structure
Rename kfd_mem_obj structure's members
Use delayed function for process tear-down

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-16 23:25:31 +03:00
Evgeny Pinchuk
5b5c4e40a3 amdkfd: Add topology module to amdkfd
This patch adds the topology module to the driver. The topology is exposed to
userspace through the sysfs.

The calls to add and remove a device to/from topology are done by the radeon
driver.

v3:

The CPU information, that is provided in the topology section of the amdkfd
driver, is extracted from the CRAT table. Unlike the CPU information located
in /sys/devices/system/cpu/cpu*, which is extracted from the SRAT table.

While the CPU information provided by the CRAT and the SRAT tables might be
identical, the node topology might be different. The SRAT table contains the
topology of CPU nodes only. The CRAT table contains the topology of CPU and GPU
nodes together (and can be interleaved). For example CPU node 1 in SRAT can be
CPU node 3 in CRAT. Furthermore it's worth to mention that the CRAT table
contains only HSA compatible nodes (nodes which are compliant with the HSA
spec).

To recap, amdkfd exposes a different kind of topology than the one exposed by
/sys/devices/system/cpu/cpu even though it may contain similar information.

v4:

The topology module doesn't support uevent handling and doesn't notify the
userspace about runtime modifications. It is up to the userspace to acquire
snapshots of the topology information created by the amdkfd and exposed
in sysfs.

The following is an example of how the topology looks on a Kaveri A10-7850K
system with amdkfd installed:

/sys/devices/virtual/kfd/kfd/
|
--- topology/
      |
      |--- generation_id
      |--- system_properties
      |--- nodes/
            |
            |--- 0/
                 |
                 |--- gpu_id
                 |--- name
                 |--- properties
                 |--- caches/
                      |
                      |--- 0/
                           |
                           |--- properties
                      |--- 1/
                           |
                           |--- properties
                      |--- 2/
                           |
                           |--- properties
                 |--- io_links/
                      |
                 |--- mem_banks/
                      |
                      |--- 0/
                           |
                           |--- properties
                      |--- 1/
                           |
                           |--- properties
                      |--- 2/
                           |
                           |--- properties
                      |--- 3/
                           |
                           |--- properties

v5:

Move amdkfd from drm/radeon/ to drm/amd/

Add a check if dev->gpu pointer is null before accessing it in the
node_show function in kfd_topology.c
This situation may occur when amdkfd is loaded and there is a GPU with a CRAT
table, but that GPU isn't supported by amdkfd

Signed-off-by: Evgeny Pinchuk <evgeny.pinchuk@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-16 21:22:32 +03:00
Oded Gabbay
4a488a7ad7 amdkfd: Add amdkfd skeleton driver
This patch adds the amdkfd skeleton driver. The driver does nothing except
define a /dev/kfd device.

It returns -ENODEV on all amdkfd IOCTLs.

v3: Move bool field to the end of structure, removed the pmc ioctls and added
a meaningful error message for ioctl error.

v5:

Create a new folder drm/amd and move amdkfd from drm/radeon/ to drm/amd/
Remove scheduler_class from kfd_priv.h as it was never used
Add skeleton implementation of the Get Version IOCTL

v6:
Update module version to the correct number and remove the "default m" from the
Kconfig file

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
2014-07-16 21:08:55 +03:00