AuxXxilium/linux_dsm_epyc7002
1
0
Fork 0
mirror of https://github.com/AuxXxilium/linux_dsm_epyc7002.git synced 2024-11-24 05:00:55 +07:00
Code Issues Actions Packages Projects Releases Wiki Activity

- ARM: GICv3 ITS emulation and various fixes. Removal of the old

Browse Source 
VGIC implementation.
 
 - s390: support for trapping software breakpoints, nested virtualization
 (vSIE), the STHYI opcode, initial extensions for CPU model support.
 
 - MIPS: support for MIPS64 hosts (32-bit guests only) and lots of cleanups,
 preliminary to this and the upcoming support for hardware virtualization
 extensions.
 
 - x86: support for execute-only mappings in nested EPT; reduced vmexit
 latency for TSC deadline timer (by about 30%) on Intel hosts; support for
 more than 255 vCPUs.
 
 - PPC: bugfixes.
 
 The ugly bit is the conflicts.  A couple of them are simple conflicts due
 to 4.7 fixes, but most of them are with other trees. There was definitely
 too much reliance on Acked-by here.  Some conflicts are for KVM patches
 where _I_ gave my Acked-by, but the worst are for this pull request's
 patches that touch files outside arch/*/kvm.  KVM submaintainers should
 probably learn to synchronize better with arch maintainers, with the
 latter providing topic branches whenever possible instead of Acked-by.
 This is what we do with arch/x86.  And I should learn to refuse pull
 requests when linux-next sends scary signals, even if that means that
 submaintainers have to rebase their branches.
 
 Anyhow, here's the list:
 
 - arch/x86/kvm/vmx.c: handle_pcommit and EXIT_REASON_PCOMMIT was removed
 by the nvdimm tree.  This tree adds handle_preemption_timer and
 EXIT_REASON_PREEMPTION_TIMER at the same place.  In general all mentions
 of pcommit have to go.
 
 There is also a conflict between a stable fix and this patch, where the
 stable fix removed the vmx_create_pml_buffer function and its call.
 
 - virt/kvm/kvm_main.c: kvm_cpu_notifier was removed by the hotplug tree.
 This tree adds kvm_io_bus_get_dev at the same place.
 
 - virt/kvm/arm/vgic.c: a few final bugfixes went into 4.7 before the
 file was completely removed for 4.8.
 
 - include/linux/irqchip/arm-gic-v3.h: this one is entirely our fault;
 this is a change that should have gone in through the irqchip tree and
 pulled by kvm-arm.  I think I would have rejected this kvm-arm pull
 request.  The KVM version is the right one, except that it lacks
 GITS_BASER_PAGES_SHIFT.
 
 - arch/powerpc: what a mess.  For the idle_book3s.S conflict, the KVM
 tree is the right one; everything else is trivial.  In this case I am
 not quite sure what went wrong.  The commit that is causing the mess
 (fd7bacbca4, "KVM: PPC: Book3S HV: Fix TB corruption in guest exit
 path on HMI interrupt", 2016-05-15) touches both arch/powerpc/kernel/
 and arch/powerpc/kvm/.  It's large, but at 396 insertions/5 deletions
 I guessed that it wasn't really possible to split it and that the 5
 deletions wouldn't conflict.  That wasn't the case.
 
 - arch/s390: also messy.  First is hypfs_diag.c where the KVM tree
 moved some code and the s390 tree patched it.  You have to reapply the
 relevant part of commits 6c22c98637, plus all of e030c1125e, to
 arch/s390/kernel/diag.c.  Or pick the linux-next conflict
 resolution from http://marc.info/?l=kvm&m=146717549531603&w=2.
 Second, there is a conflict in gmap.c between a stable fix and 4.8.
 The KVM version here is the correct one.
 
 I have pushed my resolution at refs/heads/merge-20160802 (commit
 3d1f53419842) at git://git.kernel.org/pub/scm/virt/kvm/kvm.git.
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.22 (GNU/Linux)
 
 iQEcBAABAgAGBQJXoGm7AAoJEL/70l94x66DugQIAIj703ePAFepB/fCrKHkZZia
 SGrsBdvAtNsOhr7FQ5qvvjLxiv/cv7CymeuJivX8H+4kuUHUllDzey+RPHYHD9X7
 U6n1PdCH9F15a3IXc8tDjlDdOMNIKJixYuq1UyNZMU6NFwl00+TZf9JF8A2US65b
 x/41W98ilL6nNBAsoDVmCLtPNWAqQ3lajaZELGfcqRQ9ZGKcAYOaLFXHv2YHf2XC
 qIDMf+slBGSQ66UoATnYV2gAopNlWbZ7n0vO6tE2KyvhHZ1m399aBX1+k8la/0JI
 69r+Tz7ZHUSFtmlmyByi5IAB87myy2WQHyAPwj+4vwJkDGPcl0TrupzbG7+T05Y=
 =42ti
 -----END PGP SIGNATURE-----

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull KVM updates from Paolo Bonzini:

 - ARM: GICv3 ITS emulation and various fixes.  Removal of the
   old VGIC implementation.

 - s390: support for trapping software breakpoints, nested
   virtualization (vSIE), the STHYI opcode, initial extensions
   for CPU model support.

 - MIPS: support for MIPS64 hosts (32-bit guests only) and lots
   of cleanups, preliminary to this and the upcoming support for
   hardware virtualization extensions.

 - x86: support for execute-only mappings in nested EPT; reduced
   vmexit latency for TSC deadline timer (by about 30%) on Intel
   hosts; support for more than 255 vCPUs.

 - PPC: bugfixes.

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (302 commits)
  KVM: PPC: Introduce KVM_CAP_PPC_HTM
  MIPS: Select HAVE_KVM for MIPS64_R{2,6}
  MIPS: KVM: Reset CP0_PageMask during host TLB flush
  MIPS: KVM: Fix ptr->int cast via KVM_GUEST_KSEGX()
  MIPS: KVM: Sign extend MFC0/RDHWR results
  MIPS: KVM: Fix 64-bit big endian dynamic translation
  MIPS: KVM: Fail if ebase doesn't fit in CP0_EBase
  MIPS: KVM: Use 64-bit CP0_EBase when appropriate
  MIPS: KVM: Set CP0_Status.KX on MIPS64
  MIPS: KVM: Make entry code MIPS64 friendly
  MIPS: KVM: Use kmap instead of CKSEG0ADDR()
  MIPS: KVM: Use virt_to_phys() to get commpage PFN
  MIPS: Fix definition of KSEGX() for 64-bit
  KVM: VMX: Add VMCS to CPU's loaded VMCSs before VMPTRLD
  kvm: x86: nVMX: maintain internal copy of current VMCS
  KVM: PPC: Book3S HV: Save/restore TM state in H_CEDE
  KVM: PPC: Book3S HV: Pull out TM state save/restore into separate procedures
  KVM: arm64: vgic-its: Simplify MAPI error handling
  KVM: arm64: vgic-its: Make vgic_its_cmd_handle_mapi similar to other handlers
  KVM: arm64: vgic-its: Turn device_id validation into generic ID validation
  ...
This commit is contained in:
Linus Torvalds 2016-08-02 16:11:27 -04:00
commit 221bb8a46e
167 changed files with 11790 additions and 9297 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
Documentation/virtual/kvm/api.txt
View File

@ -1482,6 +1482,11 @@ struct kvm_irq_routing_msi {
__u32 pad;
};
On x86, address_hi is ignored unless the KVM_X2APIC_API_USE_32BIT_IDS
feature of KVM_CAP_X2APIC_API capability is enabled. If it is enabled,
address_hi bits 31-8 provide bits 31-8 of the destination id. Bits 7-0 of
address_hi must be zero.
struct kvm_irq_routing_s390_adapter {
__u64 ind_addr;
__u64 summary_addr;
@ -1583,6 +1588,17 @@ struct kvm_lapic_state {
Reads the Local APIC registers and copies them into the input argument. The
data format and layout are the same as documented in the architecture manual.
If KVM_X2APIC_API_USE_32BIT_IDS feature of KVM_CAP_X2APIC_API is
enabled, then the format of APIC_ID register depends on the APIC mode
(reported by MSR_IA32_APICBASE) of its VCPU. x2APIC stores APIC ID in
the APIC_ID register (bytes 32-35). xAPIC only allows an 8-bit APIC ID
which is stored in bits 31-24 of the APIC register, or equivalently in
byte 35 of struct kvm_lapic_state's regs field. KVM_GET_LAPIC must then
be called after MSR_IA32_APICBASE has been set with KVM_SET_MSR.
If KVM_X2APIC_API_USE_32BIT_IDS feature is disabled, struct kvm_lapic_state
always uses xAPIC format.
4.58 KVM_SET_LAPIC
@ -1600,6 +1616,10 @@ struct kvm_lapic_state {
Copies the input argument into the Local APIC registers. The data format
and layout are the same as documented in the architecture manual.
The format of the APIC ID register (bytes 32-35 of struct kvm_lapic_state's
regs field) depends on the state of the KVM_CAP_X2APIC_API capability.
See the note in KVM_GET_LAPIC.
4.59 KVM_IOEVENTFD
@ -2032,6 +2052,12 @@ registers, find a list below:
MIPS | KVM_REG_MIPS_CP0_CONFIG5 | 32
MIPS | KVM_REG_MIPS_CP0_CONFIG7 | 32
MIPS | KVM_REG_MIPS_CP0_ERROREPC | 64
MIPS | KVM_REG_MIPS_CP0_KSCRATCH1 | 64
MIPS | KVM_REG_MIPS_CP0_KSCRATCH2 | 64
MIPS | KVM_REG_MIPS_CP0_KSCRATCH3 | 64
MIPS | KVM_REG_MIPS_CP0_KSCRATCH4 | 64
MIPS | KVM_REG_MIPS_CP0_KSCRATCH5 | 64
MIPS | KVM_REG_MIPS_CP0_KSCRATCH6 | 64
MIPS | KVM_REG_MIPS_COUNT_CTL | 64
MIPS | KVM_REG_MIPS_COUNT_RESUME | 64
MIPS | KVM_REG_MIPS_COUNT_HZ | 64
@ -2156,7 +2182,7 @@ after pausing the vcpu, but before it is resumed.
4.71 KVM_SIGNAL_MSI
Capability: KVM_CAP_SIGNAL_MSI
Architectures: x86
Architectures: x86 arm64
Type: vm ioctl
Parameters: struct kvm_msi (in)
Returns: >0 on delivery, 0 if guest blocked the MSI, and -1 on error
@ -2169,10 +2195,22 @@ struct kvm_msi {
__u32 address_hi;
__u32 data;
__u32 flags;
__u8 pad[16];
__u32 devid;
__u8 pad[12];
};
No flags are defined so far. The corresponding field must be 0.
flags: KVM_MSI_VALID_DEVID: devid contains a valid value
devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
for the device that wrote the MSI message.
For PCI, this is usually a BFD identifier in the lower 16 bits.
The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
the device ID. If this capability is not set, userland cannot rely on
the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
On x86, address_hi is ignored unless the KVM_CAP_X2APIC_API capability is
enabled. If it is enabled, address_hi bits 31-8 provide bits 31-8 of the
destination id. Bits 7-0 of address_hi must be zero.
4.71 KVM_CREATE_PIT2
@ -2520,6 +2558,7 @@ Parameters: struct kvm_device_attr
Returns: 0 on success, -1 on error
Errors:
ENXIO: The group or attribute is unknown/unsupported for this device
or hardware support is missing.
EPERM: The attribute cannot (currently) be accessed this way
(e.g. read-only attribute, or attribute that only makes
sense when the device is in a different state)
@ -2547,6 +2586,7 @@ Parameters: struct kvm_device_attr
Returns: 0 on success, -1 on error
Errors:
ENXIO: The group or attribute is unknown/unsupported for this device
or hardware support is missing.
Tests whether a device supports a particular attribute. A successful
return indicates the attribute is implemented. It does not necessarily
@ -3803,6 +3843,42 @@ Allows use of runtime-instrumentation introduced with zEC12 processor.
Will return -EINVAL if the machine does not support runtime-instrumentation.
Will return -EBUSY if a VCPU has already been created.
7.7 KVM_CAP_X2APIC_API
Architectures: x86
Parameters: args[0] - features that should be enabled
Returns: 0 on success, -EINVAL when args[0] contains invalid features
Valid feature flags in args[0] are
#define KVM_X2APIC_API_USE_32BIT_IDS (1ULL << 0)
#define KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK (1ULL << 1)
Enabling KVM_X2APIC_API_USE_32BIT_IDS changes the behavior of
KVM_SET_GSI_ROUTING, KVM_SIGNAL_MSI, KVM_SET_LAPIC, and KVM_GET_LAPIC,
allowing the use of 32-bit APIC IDs. See KVM_CAP_X2APIC_API in their
respective sections.
KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK must be enabled for x2APIC to work
in logical mode or with more than 255 VCPUs. Otherwise, KVM treats 0xff
as a broadcast even in x2APIC mode in order to support physical x2APIC
without interrupt remapping. This is undesirable in logical mode,
where 0xff represents CPUs 0-7 in cluster 0.
7.8 KVM_CAP_S390_USER_INSTR0
Architectures: s390
Parameters: none
With this capability enabled, all illegal instructions 0x0000 (2 bytes) will
be intercepted and forwarded to user space. User space can use this
mechanism e.g. to realize 2-byte software breakpoints. The kernel will
not inject an operating exception for these instructions, user space has
to take care of that.
This capability can be enabled dynamically even if VCPUs were already
created and are running.
8. Other capabilities.
----------------------

25
Documentation/virtual/kvm/devices/arm-vgic.txt
View File

@ -4,16 +4,22 @@ ARM Virtual Generic Interrupt Controller (VGIC)
Device types supported:
KVM_DEV_TYPE_ARM_VGIC_V2 ARM Generic Interrupt Controller v2.0
KVM_DEV_TYPE_ARM_VGIC_V3 ARM Generic Interrupt Controller v3.0
KVM_DEV_TYPE_ARM_VGIC_ITS ARM Interrupt Translation Service Controller
Only one VGIC instance may be instantiated through either this API or the
legacy KVM_CREATE_IRQCHIP api. The created VGIC will act as the VM interrupt
controller, requiring emulated user-space devices to inject interrupts to the
VGIC instead of directly to CPUs.
Only one VGIC instance of the V2/V3 types above may be instantiated through
either this API or the legacy KVM_CREATE_IRQCHIP api. The created VGIC will
act as the VM interrupt controller, requiring emulated user-space devices to
inject interrupts to the VGIC instead of directly to CPUs.
Creating a guest GICv3 device requires a host GICv3 as well.
GICv3 implementations with hardware compatibility support allow a guest GICv2
as well.
Creating a virtual ITS controller requires a host GICv3 (but does not depend
on having physical ITS controllers).
There can be multiple ITS controllers per guest, each of them has to have
a separate, non-overlapping MMIO region.
Groups:
KVM_DEV_ARM_VGIC_GRP_ADDR
Attributes:
@ -39,6 +45,13 @@ Groups:
Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
This address needs to be 64K aligned.
KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
Base address in the guest physical address space of the GICv3 ITS
control register frame. The ITS allows MSI(-X) interrupts to be
injected into guests. This extension is optional. If the kernel
does not support the ITS, the call returns -ENODEV.
Only valid for KVM_DEV_TYPE_ARM_VGIC_ITS.
This address needs to be 64K aligned and the region covers 128K.
KVM_DEV_ARM_VGIC_GRP_DIST_REGS
Attributes:
@ -109,8 +122,8 @@ Groups:
KVM_DEV_ARM_VGIC_GRP_CTRL
Attributes:
KVM_DEV_ARM_VGIC_CTRL_INIT
request the initialization of the VGIC, no additional parameter in
kvm_device_attr.addr.
request the initialization of the VGIC or ITS, no additional parameter
in kvm_device_attr.addr.
Errors:
-ENXIO: VGIC not properly configured as required prior to calling
this attribute

87
Documentation/virtual/kvm/devices/vm.txt
View File

@ -20,7 +20,8 @@ Enables Collaborative Memory Management Assist (CMMA) for the virtual machine.
1.2. ATTRIBUTE: KVM_S390_VM_MEM_CLR_CMMA
Parameters: none
Returns: 0
Returns: -EINVAL if CMMA was not enabled
0 otherwise
Clear the CMMA status for all guest pages, so any pages the guest marked
as unused are again used any may not be reclaimed by the host.
@ -85,6 +86,90 @@ Returns: -EBUSY in case 1 or more vcpus are already activated (only in write
-ENOMEM if not enough memory is available to process the ioctl
0 in case of success
2.3. ATTRIBUTE: KVM_S390_VM_CPU_MACHINE_FEAT (r/o)
Allows user space to retrieve available cpu features. A feature is available if
provided by the hardware and supported by kvm. In theory, cpu features could
even be completely emulated by kvm.
struct kvm_s390_vm_cpu_feat {
__u64 feat[16]; # Bitmap (1 = feature available), MSB 0 bit numbering
};
Parameters: address of a buffer to load the feature list from.
Returns: -EFAULT if the given address is not accessible from kernel space.
0 in case of success.
2.4. ATTRIBUTE: KVM_S390_VM_CPU_PROCESSOR_FEAT (r/w)
Allows user space to retrieve or change enabled cpu features for all VCPUs of a
VM. Features that are not available cannot be enabled.
See 2.3. for a description of the parameter struct.
Parameters: address of a buffer to store/load the feature list from.
Returns: -EFAULT if the given address is not accessible from kernel space.
-EINVAL if a cpu feature that is not available is to be enabled.
-EBUSY if at least one VCPU has already been defined.
0 in case of success.
2.5. ATTRIBUTE: KVM_S390_VM_CPU_MACHINE_SUBFUNC (r/o)
Allows user space to retrieve available cpu subfunctions without any filtering
done by a set IBC. These subfunctions are indicated to the guest VCPU via
query or "test bit" subfunctions and used e.g. by cpacf functions, plo and ptff.
A subfunction block is only valid if KVM_S390_VM_CPU_MACHINE contains the
STFL(E) bit introducing the affected instruction. If the affected instruction
indicates subfunctions via a "query subfunction", the response block is
contained in the returned struct. If the affected instruction
indicates subfunctions via a "test bit" mechanism, the subfunction codes are
contained in the returned struct in MSB 0 bit numbering.
struct kvm_s390_vm_cpu_subfunc {
u8 plo[32]; # always valid (ESA/390 feature)
u8 ptff[16]; # valid with TOD-clock steering
u8 kmac[16]; # valid with Message-Security-Assist
u8 kmc[16]; # valid with Message-Security-Assist
u8 km[16]; # valid with Message-Security-Assist
u8 kimd[16]; # valid with Message-Security-Assist
u8 klmd[16]; # valid with Message-Security-Assist
u8 pckmo[16]; # valid with Message-Security-Assist-Extension 3
u8 kmctr[16]; # valid with Message-Security-Assist-Extension 4
u8 kmf[16]; # valid with Message-Security-Assist-Extension 4
u8 kmo[16]; # valid with Message-Security-Assist-Extension 4
u8 pcc[16]; # valid with Message-Security-Assist-Extension 4
u8 ppno[16]; # valid with Message-Security-Assist-Extension 5
u8 reserved[1824]; # reserved for future instructions
};
Parameters: address of a buffer to load the subfunction blocks from.
Returns: -EFAULT if the given address is not accessible from kernel space.
0 in case of success.
2.6. ATTRIBUTE: KVM_S390_VM_CPU_PROCESSOR_SUBFUNC (r/w)
Allows user space to retrieve or change cpu subfunctions to be indicated for
all VCPUs of a VM. This attribute will only be available if kernel and
hardware support are in place.
The kernel uses the configured subfunction blocks for indication to
the guest. A subfunction block will only be used if the associated STFL(E) bit
has not been disabled by user space (so the instruction to be queried is
actually available for the guest).
As long as no data has been written, a read will fail. The IBC will be used
to determine available subfunctions in this case, this will guarantee backward
compatibility.
See 2.5. for a description of the parameter struct.
Parameters: address of a buffer to store/load the subfunction blocks from.
Returns: -EFAULT if the given address is not accessible from kernel space.
-EINVAL when reading, if there was no write yet.
-EBUSY if at least one VCPU has already been defined.
0 in case of success.
3. GROUP: KVM_S390_VM_TOD
Architectures: s390

4
Documentation/virtual/kvm/locking.txt
View File

@ -89,7 +89,7 @@ In mmu_spte_clear_track_bits():
old_spte = *spte;
/* 'if' condition is satisfied. */
if (old_spte.Accssed == 1 &&
if (old_spte.Accessed == 1 &&
old_spte.W == 0)
spte = 0ull;
on fast page fault path:
@ -102,7 +102,7 @@ In mmu_spte_clear_track_bits():
old_spte = xchg(spte, 0ull)
if (old_spte.Accssed == 1)
if (old_spte.Accessed == 1)
kvm_set_pfn_accessed(spte.pfn);
if (old_spte.Dirty == 1)
kvm_set_pfn_dirty(spte.pfn);

2
arch/arm/include/asm/kvm_asm.h
View File

@ -66,6 +66,8 @@ extern void __kvm_tlb_flush_vmid(struct kvm *kvm);
extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
extern void __init_stage2_translation(void);
extern void __kvm_hyp_reset(unsigned long);
#endif
#endif /* __ARM_KVM_ASM_H__ */

27
arch/arm/include/asm/kvm_host.h
View File

@ -241,8 +241,7 @@ int kvm_arm_coproc_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *);
int handle_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
int exception_index);
static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr,
phys_addr_t pgd_ptr,
static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
unsigned long hyp_stack_ptr,
unsigned long vector_ptr)
{
@ -251,18 +250,13 @@ static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr,
* code. The init code doesn't need to preserve these
* registers as r0-r3 are already callee saved according to
* the AAPCS.
* Note that we slightly misuse the prototype by casing the
* Note that we slightly misuse the prototype by casting the
* stack pointer to a void *.
*
* We don't have enough registers to perform the full init in
* one go. Install the boot PGD first, and then install the
* runtime PGD, stack pointer and vectors. The PGDs are always
* passed as the third argument, in order to be passed into
* r2-r3 to the init code (yes, this is compliant with the
* PCS!).
*/
kvm_call_hyp(NULL, 0, boot_pgd_ptr);
* The PGDs are always passed as the third argument, in order
* to be passed into r2-r3 to the init code (yes, this is
* compliant with the PCS!).
*/
kvm_call_hyp((void*)hyp_stack_ptr, vector_ptr, pgd_ptr);
}
@ -272,16 +266,13 @@ static inline void __cpu_init_stage2(void)
kvm_call_hyp(__init_stage2_translation);
}
static inline void __cpu_reset_hyp_mode(phys_addr_t boot_pgd_ptr,
static inline void __cpu_reset_hyp_mode(unsigned long vector_ptr,
phys_addr_t phys_idmap_start)
{
/*
* TODO
* kvm_call_reset(boot_pgd_ptr, phys_idmap_start);
*/
kvm_call_hyp((void *)virt_to_idmap(__kvm_hyp_reset), vector_ptr);
}
static inline int kvm_arch_dev_ioctl_check_extension(long ext)
static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
{
return 0;
}

3
arch/arm/include/asm/kvm_hyp.h
View File

@ -25,9 +25,6 @@
#define __hyp_text __section(.hyp.text) notrace
#define kern_hyp_va(v) (v)
#define hyp_kern_va(v) (v)
#define __ACCESS_CP15(CRn, Op1, CRm, Op2) \
"mrc", "mcr", __stringify(p15, Op1, %0, CRn, CRm, Op2), u32
#define __ACCESS_CP15_64(Op1, CRm) \

15
arch/arm/include/asm/kvm_mmu.h
View File

@ -26,16 +26,7 @@
* We directly use the kernel VA for the HYP, as we can directly share
* the mapping (HTTBR "covers" TTBR1).
*/
#define HYP_PAGE_OFFSET_MASK UL(~0)
#define HYP_PAGE_OFFSET PAGE_OFFSET
#define KERN_TO_HYP(kva) (kva)
/*
* Our virtual mapping for the boot-time MMU-enable code. Must be
* shared across all the page-tables. Conveniently, we use the vectors
* page, where no kernel data will ever be shared with HYP.
*/
#define TRAMPOLINE_VA UL(CONFIG_VECTORS_BASE)
#define kern_hyp_va(kva) (kva)
/*
* KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
@ -49,9 +40,8 @@
#include <asm/pgalloc.h>
#include <asm/stage2_pgtable.h>
int create_hyp_mappings(void *from, void *to);
int create_hyp_mappings(void *from, void *to, pgprot_t prot);
int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
void free_boot_hyp_pgd(void);
void free_hyp_pgds(void);
void stage2_unmap_vm(struct kvm *kvm);
@ -65,7 +55,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run);
void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);
phys_addr_t kvm_mmu_get_httbr(void);
phys_addr_t kvm_mmu_get_boot_httbr(void);
phys_addr_t kvm_get_idmap_vector(void);
phys_addr_t kvm_get_idmap_start(void);
int kvm_mmu_init(void);

4
arch/arm/include/asm/pgtable.h
View File

@ -97,7 +97,9 @@ extern pgprot_t pgprot_s2_device;
#define PAGE_READONLY_EXEC _MOD_PROT(pgprot_user, L_PTE_USER | L_PTE_RDONLY)
#define PAGE_KERNEL _MOD_PROT(pgprot_kernel, L_PTE_XN)
#define PAGE_KERNEL_EXEC pgprot_kernel
#define PAGE_HYP _MOD_PROT(pgprot_kernel, L_PTE_HYP)
#define PAGE_HYP _MOD_PROT(pgprot_kernel, L_PTE_HYP | L_PTE_XN)
#define PAGE_HYP_EXEC _MOD_PROT(pgprot_kernel, L_PTE_HYP | L_PTE_RDONLY)
#define PAGE_HYP_RO _MOD_PROT(pgprot_kernel, L_PTE_HYP | L_PTE_RDONLY | L_PTE_XN)
#define PAGE_HYP_DEVICE _MOD_PROT(pgprot_hyp_device, L_PTE_HYP)
#define PAGE_S2 _MOD_PROT(pgprot_s2, L_PTE_S2_RDONLY)
#define PAGE_S2_DEVICE _MOD_PROT(pgprot_s2_device, L_PTE_S2_RDONLY)

4
arch/arm/include/asm/virt.h
View File

@ -80,6 +80,10 @@ static inline bool is_kernel_in_hyp_mode(void)
return false;
}
/* The section containing the hypervisor idmap text */
extern char __hyp_idmap_text_start[];
extern char __hyp_idmap_text_end[];
/* The section containing the hypervisor text */
extern char __hyp_text_start[];
extern char __hyp_text_end[];

7
arch/arm/kvm/Kconfig
View File

@ -46,13 +46,6 @@ config KVM_ARM_HOST
---help---
Provides host support for ARM processors.
config KVM_NEW_VGIC
bool "New VGIC implementation"
depends on KVM
default y
---help---
uses the new VGIC implementation
source drivers/vhost/Kconfig
endif # VIRTUALIZATION

6
arch/arm/kvm/Makefile
View File

@ -22,7 +22,6 @@ obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o
ifeq ($(CONFIG_KVM_NEW_VGIC),y)
obj-y += $(KVM)/arm/vgic/vgic.o
obj-y += $(KVM)/arm/vgic/vgic-init.o
obj-y += $(KVM)/arm/vgic/vgic-irqfd.o
@ -30,9 +29,4 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
obj-y += $(KVM)/arm/vgic/vgic-mmio.o
obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
else
obj-y += $(KVM)/arm/vgic.o
obj-y += $(KVM)/arm/vgic-v2.o
obj-y += $(KVM)/arm/vgic-v2-emul.o
endif
obj-y += $(KVM)/arm/arch_timer.o

46
arch/arm/kvm/arm.c
View File

@ -20,6 +20,7 @@
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/kvm_host.h>
#include <linux/list.h>
#include <linux/module.h>
#include <linux/vmalloc.h>
#include <linux/fs.h>
@ -122,7 +123,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
if (ret)
goto out_fail_alloc;
ret = create_hyp_mappings(kvm, kvm + 1);
ret = create_hyp_mappings(kvm, kvm + 1, PAGE_HYP);
if (ret)
goto out_free_stage2_pgd;
@ -201,7 +202,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
r = KVM_MAX_VCPUS;
break;
default:
r = kvm_arch_dev_ioctl_check_extension(ext);
r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
break;
}
return r;
@ -239,7 +240,7 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
if (err)
goto free_vcpu;
err = create_hyp_mappings(vcpu, vcpu + 1);
err = create_hyp_mappings(vcpu, vcpu + 1, PAGE_HYP);
if (err)
goto vcpu_uninit;
@ -377,7 +378,7 @@ void force_vm_exit(const cpumask_t *mask)
/**
* need_new_vmid_gen - check that the VMID is still valid
* @kvm: The VM's VMID to checkt
* @kvm: The VM's VMID to check
*
* return true if there is a new generation of VMIDs being used
*
@ -616,7 +617,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
* Enter the guest
*/
trace_kvm_entry(*vcpu_pc(vcpu));
__kvm_guest_enter();
guest_enter_irqoff();
vcpu->mode = IN_GUEST_MODE;
ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);
@ -642,14 +643,14 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
local_irq_enable();
/*
* We do local_irq_enable() before calling kvm_guest_exit() so
* We do local_irq_enable() before calling guest_exit() so
* that if a timer interrupt hits while running the guest we
* account that tick as being spent in the guest. We enable
* preemption after calling kvm_guest_exit() so that if we get
* preemption after calling guest_exit() so that if we get
* preempted we make sure ticks after that is not counted as
* guest time.
*/
kvm_guest_exit();
guest_exit();
trace_kvm_exit(ret, kvm_vcpu_trap_get_class(vcpu), *vcpu_pc(vcpu));
/*
@ -1039,7 +1040,6 @@ long kvm_arch_vm_ioctl(struct file *filp,
static void cpu_init_hyp_mode(void *dummy)
{
phys_addr_t boot_pgd_ptr;
phys_addr_t pgd_ptr;
unsigned long hyp_stack_ptr;
unsigned long stack_page;
@ -1048,13 +1048,12 @@ static void cpu_init_hyp_mode(void *dummy)
/* Switch from the HYP stub to our own HYP init vector */
__hyp_set_vectors(kvm_get_idmap_vector());
boot_pgd_ptr = kvm_mmu_get_boot_httbr();
pgd_ptr = kvm_mmu_get_httbr();
stack_page = __this_cpu_read(kvm_arm_hyp_stack_page);
hyp_stack_ptr = stack_page + PAGE_SIZE;
vector_ptr = (unsigned long)kvm_ksym_ref(__kvm_hyp_vector);
__cpu_init_hyp_mode(boot_pgd_ptr, pgd_ptr, hyp_stack_ptr, vector_ptr);
__cpu_init_hyp_mode(pgd_ptr, hyp_stack_ptr, vector_ptr);
__cpu_init_stage2();
kvm_arm_init_debug();
@ -1076,15 +1075,9 @@ static void cpu_hyp_reinit(void)
static void cpu_hyp_reset(void)
{
phys_addr_t boot_pgd_ptr;
phys_addr_t phys_idmap_start;
if (!is_kernel_in_hyp_mode()) {
boot_pgd_ptr = kvm_mmu_get_boot_httbr();
phys_idmap_start = kvm_get_idmap_start();
__cpu_reset_hyp_mode(boot_pgd_ptr, phys_idmap_start);
}
if (!is_kernel_in_hyp_mode())
__cpu_reset_hyp_mode(hyp_default_vectors,
kvm_get_idmap_start());
}
static void _kvm_arch_hardware_enable(void *discard)
@ -1294,14 +1287,14 @@ static int init_hyp_mode(void)
* Map the Hyp-code called directly from the host
*/
err = create_hyp_mappings(kvm_ksym_ref(__hyp_text_start),
kvm_ksym_ref(__hyp_text_end));
kvm_ksym_ref(__hyp_text_end), PAGE_HYP_EXEC);
if (err) {
kvm_err("Cannot map world-switch code\n");
goto out_err;
}
err = create_hyp_mappings(kvm_ksym_ref(__start_rodata),
kvm_ksym_ref(__end_rodata));
kvm_ksym_ref(__end_rodata), PAGE_HYP_RO);
if (err) {
kvm_err("Cannot map rodata section\n");
goto out_err;
@ -1312,7 +1305,8 @@ static int init_hyp_mode(void)
*/
for_each_possible_cpu(cpu) {
char *stack_page = (char *)per_cpu(kvm_arm_hyp_stack_page, cpu);
err = create_hyp_mappings(stack_page, stack_page + PAGE_SIZE);
err = create_hyp_mappings(stack_page, stack_page + PAGE_SIZE,
PAGE_HYP);
if (err) {
kvm_err("Cannot map hyp stack\n");
@ -1324,7 +1318,7 @@ static int init_hyp_mode(void)
kvm_cpu_context_t *cpu_ctxt;
cpu_ctxt = per_cpu_ptr(kvm_host_cpu_state, cpu);
err = create_hyp_mappings(cpu_ctxt, cpu_ctxt + 1);
err = create_hyp_mappings(cpu_ctxt, cpu_ctxt + 1, PAGE_HYP);
if (err) {
kvm_err("Cannot map host CPU state: %d\n", err);
@ -1332,10 +1326,6 @@ static int init_hyp_mode(void)
}
}
#ifndef CONFIG_HOTPLUG_CPU
free_boot_hyp_pgd();
#endif
/* set size of VMID supported by CPU */
kvm_vmid_bits = kvm_get_vmid_bits();
kvm_info("%d-bit VMID\n", kvm_vmid_bits);

2
arch/arm/kvm/emulate.c
View File

@ -210,7 +210,7 @@ bool kvm_condition_valid(struct kvm_vcpu *vcpu)
* @vcpu: The VCPU pointer
*
* When exceptions occur while instructions are executed in Thumb IF-THEN
* blocks, the ITSTATE field of the CPSR is not advanved (updated), so we have
* blocks, the ITSTATE field of the CPSR is not advanced (updated), so we have
* to do this little bit of work manually. The fields map like this:
*
* IT[7:0] -> CPSR[26:25],CPSR[15:10]

2
arch/arm/kvm/guest.c
View File

@ -182,7 +182,7 @@ unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
/**
* kvm_arm_copy_reg_indices - get indices of all registers.
*
* We do core registers right here, then we apppend coproc regs.
* We do core registers right here, then we append coproc regs.
*/
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
{

62
arch/arm/kvm/init.S
View File

@ -32,23 +32,13 @@
* r2,r3 = Hypervisor pgd pointer
*
* The init scenario is:
* - We jump in HYP with four parameters: boot HYP pgd, runtime HYP pgd,
* runtime stack, runtime vectors
* - Enable the MMU with the boot pgd
* - Jump to a target into the trampoline page (remember, this is the same
* physical page!)
* - Now switch to the runtime pgd (same VA, and still the same physical
* page!)
* - We jump in HYP with 3 parameters: runtime HYP pgd, runtime stack,
* runtime vectors
* - Invalidate TLBs
* - Set stack and vectors
* - Setup the page tables
* - Enable the MMU
* - Profit! (or eret, if you only care about the code).
*
* As we only have four registers available to pass parameters (and we
* need six), we split the init in two phases:
* - Phase 1: r0 = 0, r1 = 0, r2,r3 contain the boot PGD.
* Provides the basic HYP init, and enable the MMU.
* - Phase 2: r0 = ToS, r1 = vectors, r2,r3 contain the runtime PGD.
* Switches to the runtime PGD, set stack and vectors.
*/
.text
@ -68,8 +58,11 @@ __kvm_hyp_init:
W(b) .
__do_hyp_init:
cmp r0, #0 @ We have a SP?
bne phase2 @ Yes, second stage init
@ Set stack pointer
mov sp, r0
@ Set HVBAR to point to the HYP vectors
mcr p15, 4, r1, c12, c0, 0 @ HVBAR
@ Set the HTTBR to point to the hypervisor PGD pointer passed
mcrr p15, 4, rr_lo_hi(r2, r3), c2
@ -114,34 +107,25 @@ __do_hyp_init:
THUMB( ldr r2, =(HSCTLR_M | HSCTLR_A | HSCTLR_TE) )
orr r1, r1, r2
orr r0, r0, r1
isb
mcr p15, 4, r0, c1, c0, 0 @ HSCR
@ End of init phase-1
eret
phase2:
@ Set stack pointer
mov sp, r0
@ Set HVBAR to point to the HYP vectors
mcr p15, 4, r1, c12, c0, 0 @ HVBAR
@ Jump to the trampoline page
ldr r0, =TRAMPOLINE_VA
adr r1, target
bfi r0, r1, #0, #PAGE_SHIFT
ret r0
target: @ We're now in the trampoline code, switch page tables
mcrr p15, 4, rr_lo_hi(r2, r3), c2
isb
@ Invalidate the old TLBs
mcr p15, 4, r0, c8, c7, 0 @ TLBIALLH
dsb ish
eret
@ r0 : stub vectors address
ENTRY(__kvm_hyp_reset)
/* We're now in idmap, disable MMU */
mrc p15, 4, r1, c1, c0, 0 @ HSCTLR
ldr r2, =(HSCTLR_M | HSCTLR_A | HSCTLR_C | HSCTLR_I)
bic r1, r1, r2
mcr p15, 4, r1, c1, c0, 0 @ HSCTLR
/* Install stub vectors */
mcr p15, 4, r0, c12, c0, 0 @ HVBAR
isb
eret
ENDPROC(__kvm_hyp_reset)
.ltorg

144
arch/arm/kvm/mmu.c
View File

@ -32,8 +32,6 @@
#include "trace.h"
extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[];
static pgd_t *boot_hyp_pgd;
static pgd_t *hyp_pgd;
static pgd_t *merged_hyp_pgd;
@ -483,28 +481,6 @@ static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
} while (pgd++, addr = next, addr != end);
}
/**
* free_boot_hyp_pgd - free HYP boot page tables
*
* Free the HYP boot page tables. The bounce page is also freed.
*/
void free_boot_hyp_pgd(void)
{
mutex_lock(&kvm_hyp_pgd_mutex);
if (boot_hyp_pgd) {
unmap_hyp_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
unmap_hyp_range(boot_hyp_pgd, TRAMPOLINE_VA, PAGE_SIZE);
free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
boot_hyp_pgd = NULL;
}
if (hyp_pgd)
unmap_hyp_range(hyp_pgd, TRAMPOLINE_VA, PAGE_SIZE);
mutex_unlock(&kvm_hyp_pgd_mutex);
}
/**
* free_hyp_pgds - free Hyp-mode page tables
*
@ -519,15 +495,20 @@ void free_hyp_pgds(void)
{
unsigned long addr;
free_boot_hyp_pgd();
mutex_lock(&kvm_hyp_pgd_mutex);
if (boot_hyp_pgd) {
unmap_hyp_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
boot_hyp_pgd = NULL;
}
if (hyp_pgd) {
unmap_hyp_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
for (addr = PAGE_OFFSET; virt_addr_valid(addr); addr += PGDIR_SIZE)
unmap_hyp_range(hyp_pgd, KERN_TO_HYP(addr), PGDIR_SIZE);
unmap_hyp_range(hyp_pgd, kern_hyp_va(addr), PGDIR_SIZE);
for (addr = VMALLOC_START; is_vmalloc_addr((void*)addr); addr += PGDIR_SIZE)
unmap_hyp_range(hyp_pgd, KERN_TO_HYP(addr), PGDIR_SIZE);
unmap_hyp_range(hyp_pgd, kern_hyp_va(addr), PGDIR_SIZE);
free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
hyp_pgd = NULL;
@ -679,17 +660,18 @@ static phys_addr_t kvm_kaddr_to_phys(void *kaddr)
* create_hyp_mappings - duplicate a kernel virtual address range in Hyp mode
* @from: The virtual kernel start address of the range
* @to: The virtual kernel end address of the range (exclusive)
* @prot: The protection to be applied to this range
*
* The same virtual address as the kernel virtual address is also used
* in Hyp-mode mapping (modulo HYP_PAGE_OFFSET) to the same underlying
* physical pages.
*/
int create_hyp_mappings(void *from, void *to)
int create_hyp_mappings(void *from, void *to, pgprot_t prot)
{
phys_addr_t phys_addr;
unsigned long virt_addr;
unsigned long start = KERN_TO_HYP((unsigned long)from);
unsigned long end = KERN_TO_HYP((unsigned long)to);
unsigned long start = kern_hyp_va((unsigned long)from);
unsigned long end = kern_hyp_va((unsigned long)to);
if (is_kernel_in_hyp_mode())
return 0;
@ -704,7 +686,7 @@ int create_hyp_mappings(void *from, void *to)
err = __create_hyp_mappings(hyp_pgd, virt_addr,
virt_addr + PAGE_SIZE,
__phys_to_pfn(phys_addr),
PAGE_HYP);
prot);
if (err)
return err;
}
@ -723,8 +705,8 @@ int create_hyp_mappings(void *from, void *to)
*/
int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
{
unsigned long start = KERN_TO_HYP((unsigned long)from);
unsigned long end = KERN_TO_HYP((unsigned long)to);
unsigned long start = kern_hyp_va((unsigned long)from);
unsigned long end = kern_hyp_va((unsigned long)to);
if (is_kernel_in_hyp_mode())
return 0;
@ -1687,14 +1669,6 @@ phys_addr_t kvm_mmu_get_httbr(void)
return virt_to_phys(hyp_pgd);
}
phys_addr_t kvm_mmu_get_boot_httbr(void)
{
if (__kvm_cpu_uses_extended_idmap())
return virt_to_phys(merged_hyp_pgd);
else
return virt_to_phys(boot_hyp_pgd);
}
phys_addr_t kvm_get_idmap_vector(void)
{
return hyp_idmap_vector;
@ -1705,6 +1679,22 @@ phys_addr_t kvm_get_idmap_start(void)
return hyp_idmap_start;
}
static int kvm_map_idmap_text(pgd_t *pgd)
{
int err;
/* Create the idmap in the boot page tables */
err = __create_hyp_mappings(pgd,
hyp_idmap_start, hyp_idmap_end,
__phys_to_pfn(hyp_idmap_start),
PAGE_HYP_EXEC);
if (err)
kvm_err("Failed to idmap %lx-%lx\n",
hyp_idmap_start, hyp_idmap_end);
return err;
}
int kvm_mmu_init(void)
{
int err;
@ -1719,28 +1709,41 @@ int kvm_mmu_init(void)
*/
BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK);
hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, hyp_pgd_order);
boot_hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, hyp_pgd_order);
kvm_info("IDMAP page: %lx\n", hyp_idmap_start);
kvm_info("HYP VA range: %lx:%lx\n",
kern_hyp_va(PAGE_OFFSET), kern_hyp_va(~0UL));
if (!hyp_pgd || !boot_hyp_pgd) {
if (hyp_idmap_start >= kern_hyp_va(PAGE_OFFSET) &&
hyp_idmap_start < kern_hyp_va(~0UL)) {
/*
* The idmap page is intersecting with the VA space,
* it is not safe to continue further.
*/
kvm_err("IDMAP intersecting with HYP VA, unable to continue\n");
err = -EINVAL;
goto out;
}
hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, hyp_pgd_order);
if (!hyp_pgd) {
kvm_err("Hyp mode PGD not allocated\n");
err = -ENOMEM;
goto out;
}
/* Create the idmap in the boot page tables */
err = __create_hyp_mappings(boot_hyp_pgd,
hyp_idmap_start, hyp_idmap_end,
__phys_to_pfn(hyp_idmap_start),
PAGE_HYP);
if (err) {
kvm_err("Failed to idmap %lx-%lx\n",
hyp_idmap_start, hyp_idmap_end);
goto out;
}
if (__kvm_cpu_uses_extended_idmap()) {
boot_hyp_pgd = (pgd_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO,
hyp_pgd_order);
if (!boot_hyp_pgd) {
kvm_err("Hyp boot PGD not allocated\n");
err = -ENOMEM;
goto out;
}
err = kvm_map_idmap_text(boot_hyp_pgd);
if (err)
goto out;
merged_hyp_pgd = (pgd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
if (!merged_hyp_pgd) {
kvm_err("Failed to allocate extra HYP pgd\n");
@ -1748,29 +1751,10 @@ int kvm_mmu_init(void)
}
__kvm_extend_hypmap(boot_hyp_pgd, hyp_pgd, merged_hyp_pgd,
hyp_idmap_start);
return 0;
}
/* Map the very same page at the trampoline VA */
err = __create_hyp_mappings(boot_hyp_pgd,
TRAMPOLINE_VA, TRAMPOLINE_VA + PAGE_SIZE,
__phys_to_pfn(hyp_idmap_start),
PAGE_HYP);
if (err) {
kvm_err("Failed to map trampoline @%lx into boot HYP pgd\n",
TRAMPOLINE_VA);
goto out;
}
/* Map the same page again into the runtime page tables */
err = __create_hyp_mappings(hyp_pgd,
TRAMPOLINE_VA, TRAMPOLINE_VA + PAGE_SIZE,
__phys_to_pfn(hyp_idmap_start),
PAGE_HYP);
if (err) {
kvm_err("Failed to map trampoline @%lx into runtime HYP pgd\n",
TRAMPOLINE_VA);
goto out;
} else {
err = kvm_map_idmap_text(hyp_pgd);
if (err)
goto out;
}
return 0;

2
arch/arm/kvm/reset.c
View File

@ -52,7 +52,7 @@ static const struct kvm_irq_level cortexa_vtimer_irq = {
* @vcpu: The VCPU pointer
*
* This function finds the right table above and sets the registers on the
* virtual CPU struct to their architectually defined reset values.
* virtual CPU struct to their architecturally defined reset values.
*/
int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
{

3
arch/arm64/include/asm/cpufeature.h
View File

@ -36,8 +36,9 @@
#define ARM64_HAS_VIRT_HOST_EXTN 11
#define ARM64_WORKAROUND_CAVIUM_27456 12
#define ARM64_HAS_32BIT_EL0 13
#define ARM64_HYP_OFFSET_LOW 14
#define ARM64_NCAPS 14
#define ARM64_NCAPS 15
#ifndef __ASSEMBLY__

2
arch/arm64/include/asm/kvm_arm.h
View File

@ -178,7 +178,7 @@
/* Hyp System Trap Register */
#define HSTR_EL2_T(x) (1 << x)
/* Hyp Coproccessor Trap Register Shifts */
/* Hyp Coprocessor Trap Register Shifts */
#define CPTR_EL2_TFP_SHIFT 10
/* Hyp Coprocessor Trap Register */

19
arch/arm64/include/asm/kvm_host.h
View File

@ -47,8 +47,7 @@
int __attribute_const__ kvm_target_cpu(void);
int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
int kvm_arch_dev_ioctl_check_extension(long ext);
unsigned long kvm_hyp_reset_entry(void);
int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
void __extended_idmap_trampoline(phys_addr_t boot_pgd, phys_addr_t idmap_start);
struct kvm_arch {
@ -348,8 +347,7 @@ int kvm_perf_teardown(void);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr,
phys_addr_t pgd_ptr,
static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
unsigned long hyp_stack_ptr,
unsigned long vector_ptr)
{
@ -357,19 +355,14 @@ static inline void __cpu_init_hyp_mode(phys_addr_t boot_pgd_ptr,
* Call initialization code, and switch to the full blown
* HYP code.
*/
__kvm_call_hyp((void *)boot_pgd_ptr, pgd_ptr,
hyp_stack_ptr, vector_ptr);
__kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
}
static inline void __cpu_reset_hyp_mode(phys_addr_t boot_pgd_ptr,
void __kvm_hyp_teardown(void);
static inline void __cpu_reset_hyp_mode(unsigned long vector_ptr,
phys_addr_t phys_idmap_start)
{
/*
* Call reset code, and switch back to stub hyp vectors.
* Uses __kvm_call_hyp() to avoid kaslr's kvm_ksym_ref() translation.
*/
__kvm_call_hyp((void *)kvm_hyp_reset_entry(),
boot_pgd_ptr, phys_idmap_start);
kvm_call_hyp(__kvm_hyp_teardown, phys_idmap_start);
}
static inline void kvm_arch_hardware_unsetup(void) {}

23
arch/arm64/include/asm/kvm_hyp.h
View File

@ -25,29 +25,6 @@
#define __hyp_text __section(.hyp.text) notrace
static inline unsigned long __kern_hyp_va(unsigned long v)
{
asm volatile(ALTERNATIVE("and %0, %0, %1",
"nop",
ARM64_HAS_VIRT_HOST_EXTN)
: "+r" (v) : "i" (HYP_PAGE_OFFSET_MASK));
return v;
}
#define kern_hyp_va(v) (typeof(v))(__kern_hyp_va((unsigned long)(v)))
static inline unsigned long __hyp_kern_va(unsigned long v)
{
u64 offset = PAGE_OFFSET - HYP_PAGE_OFFSET;
asm volatile(ALTERNATIVE("add %0, %0, %1",
"nop",
ARM64_HAS_VIRT_HOST_EXTN)
: "+r" (v) : "r" (offset));
return v;
}
#define hyp_kern_va(v) (typeof(v))(__hyp_kern_va((unsigned long)(v)))
#define read_sysreg_elx(r,nvh,vh) \
({ \
u64 reg; \

96
arch/arm64/include/asm/kvm_mmu.h
View File

@ -29,21 +29,48 @@
*
* Instead, give the HYP mode its own VA region at a fixed offset from
* the kernel by just masking the top bits (which are all ones for a
* kernel address).
* kernel address). We need to find out how many bits to mask.
*
* ARMv8.1 (using VHE) does have a TTBR1_EL2, and doesn't use these
* macros (the entire kernel runs at EL2).
* We want to build a set of page tables that cover both parts of the
* idmap (the trampoline page used to initialize EL2), and our normal
* runtime VA space, at the same time.
*
* Given that the kernel uses VA_BITS for its entire address space,
* and that half of that space (VA_BITS - 1) is used for the linear
* mapping, we can also limit the EL2 space to (VA_BITS - 1).
*
* The main question is "Within the VA_BITS space, does EL2 use the
* top or the bottom half of that space to shadow the kernel's linear
* mapping?". As we need to idmap the trampoline page, this is
* determined by the range in which this page lives.
*
* If the page is in the bottom half, we have to use the top half. If
* the page is in the top half, we have to use the bottom half:
*
* T = __virt_to_phys(__hyp_idmap_text_start)
* if (T & BIT(VA_BITS - 1))
* HYP_VA_MIN = 0 //idmap in upper half
* else
* HYP_VA_MIN = 1 << (VA_BITS - 1)
* HYP_VA_MAX = HYP_VA_MIN + (1 << (VA_BITS - 1)) - 1
*
* This of course assumes that the trampoline page exists within the
* VA_BITS range. If it doesn't, then it means we're in the odd case
* where the kernel idmap (as well as HYP) uses more levels than the
* kernel runtime page tables (as seen when the kernel is configured
* for 4k pages, 39bits VA, and yet memory lives just above that
* limit, forcing the idmap to use 4 levels of page tables while the
* kernel itself only uses 3). In this particular case, it doesn't
* matter which side of VA_BITS we use, as we're guaranteed not to
* conflict with anything.
*
* When using VHE, there are no separate hyp mappings and all KVM
* functionality is already mapped as part of the main kernel
* mappings, and none of this applies in that case.
*/
#define HYP_PAGE_OFFSET_SHIFT VA_BITS
#define HYP_PAGE_OFFSET_MASK ((UL(1) << HYP_PAGE_OFFSET_SHIFT) - 1)
#define HYP_PAGE_OFFSET (PAGE_OFFSET & HYP_PAGE_OFFSET_MASK)
/*
* Our virtual mapping for the idmap-ed MMU-enable code. Must be
* shared across all the page-tables. Conveniently, we use the last
* possible page, where no kernel mapping will ever exist.
*/
#define TRAMPOLINE_VA (HYP_PAGE_OFFSET_MASK & PAGE_MASK)
#define HYP_PAGE_OFFSET_HIGH_MASK ((UL(1) << VA_BITS) - 1)
#define HYP_PAGE_OFFSET_LOW_MASK ((UL(1) << (VA_BITS - 1)) - 1)
#ifdef __ASSEMBLY__
@ -53,13 +80,33 @@
/*
* Convert a kernel VA into a HYP VA.
* reg: VA to be converted.
*
* This generates the following sequences:
* - High mask:
* and x0, x0, #HYP_PAGE_OFFSET_HIGH_MASK
* nop
* - Low mask:
* and x0, x0, #HYP_PAGE_OFFSET_HIGH_MASK
* and x0, x0, #HYP_PAGE_OFFSET_LOW_MASK
* - VHE:
* nop
* nop
*
* The "low mask" version works because the mask is a strict subset of
* the "high mask", hence performing the first mask for nothing.
* Should be completely invisible on any viable CPU.
*/
.macro kern_hyp_va reg
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
and \reg, \reg, #HYP_PAGE_OFFSET_MASK
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
and \reg, \reg, #HYP_PAGE_OFFSET_HIGH_MASK
alternative_else
nop
alternative_endif
alternative_if_not ARM64_HYP_OFFSET_LOW
nop
alternative_else
and \reg, \reg, #HYP_PAGE_OFFSET_LOW_MASK
alternative_endif
.endm
#else
@ -70,7 +117,22 @@ alternative_endif
#include <asm/mmu_context.h>
#include <asm/pgtable.h>
#define KERN_TO_HYP(kva) ((unsigned long)kva - PAGE_OFFSET + HYP_PAGE_OFFSET)
static inline unsigned long __kern_hyp_va(unsigned long v)
{
asm volatile(ALTERNATIVE("and %0, %0, %1",
"nop",
ARM64_HAS_VIRT_HOST_EXTN)
: "+r" (v)
: "i" (HYP_PAGE_OFFSET_HIGH_MASK));
asm volatile(ALTERNATIVE("nop",
"and %0, %0, %1",
ARM64_HYP_OFFSET_LOW)
: "+r" (v)
: "i" (HYP_PAGE_OFFSET_LOW_MASK));
return v;
}
#define kern_hyp_va(v) (typeof(v))(__kern_hyp_va((unsigned long)(v)))
/*
* We currently only support a 40bit IPA.
@ -81,9 +143,8 @@ alternative_endif
#include <asm/stage2_pgtable.h>
int create_hyp_mappings(void *from, void *to);
int create_hyp_mappings(void *from, void *to, pgprot_t prot);
int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
void free_boot_hyp_pgd(void);
void free_hyp_pgds(void);
void stage2_unmap_vm(struct kvm *kvm);
@ -97,7 +158,6 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run);
void kvm_mmu_free_memory_caches(struct kvm_vcpu *vcpu);
phys_addr_t kvm_mmu_get_httbr(void);
phys_addr_t kvm_mmu_get_boot_httbr(void);
phys_addr_t kvm_get_idmap_vector(void);
phys_addr_t kvm_get_idmap_start(void);
int kvm_mmu_init(void);

1
arch/arm64/include/asm/pgtable-hwdef.h
View File

@ -164,6 +164,7 @@
#define PTE_CONT (_AT(pteval_t, 1) << 52) /* Contiguous range */
#define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */
#define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */
#define PTE_HYP_XN (_AT(pteval_t, 1) << 54) /* HYP XN */
/*
* AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registers).

4
arch/arm64/include/asm/pgtable-prot.h
View File

@ -55,7 +55,9 @@
#define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE)
#define PAGE_KERNEL_EXEC_CONT __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_CONT)
#define PAGE_HYP __pgprot(_PAGE_DEFAULT | PTE_HYP)
#define PAGE_HYP __pgprot(_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)
#define PAGE_HYP_EXEC __pgprot(_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)
#define PAGE_HYP_RO __pgprot(_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)
#define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP)
#define PAGE_S2 __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)

4
arch/arm64/include/asm/virt.h
View File

@ -87,6 +87,10 @@ extern void verify_cpu_run_el(void);
static inline void verify_cpu_run_el(void) {}
#endif
/* The section containing the hypervisor idmap text */
extern char __hyp_idmap_text_start[];
extern char __hyp_idmap_text_end[];
/* The section containing the hypervisor text */
extern char __hyp_text_start[];
extern char __hyp_text_end[];

2
arch/arm64/include/uapi/asm/kvm.h
View File

@ -87,9 +87,11 @@ struct kvm_regs {
/* Supported VGICv3 address types */
#define KVM_VGIC_V3_ADDR_TYPE_DIST 2
#define KVM_VGIC_V3_ADDR_TYPE_REDIST 3
#define KVM_VGIC_ITS_ADDR_TYPE 4
#define KVM_VGIC_V3_DIST_SIZE SZ_64K
#define KVM_VGIC_V3_REDIST_SIZE (2 * SZ_64K)
#define KVM_VGIC_V3_ITS_SIZE (2 * SZ_64K)
#define KVM_ARM_VCPU_POWER_OFF 0 /* CPU is started in OFF state */
#define KVM_ARM_VCPU_EL1_32BIT 1 /* CPU running a 32bit VM */

19
arch/arm64/kernel/cpufeature.c
View File

@ -726,6 +726,19 @@ static bool runs_at_el2(const struct arm64_cpu_capabilities *entry, int __unused
return is_kernel_in_hyp_mode();
}
static bool hyp_offset_low(const struct arm64_cpu_capabilities *entry,
int __unused)
{
phys_addr_t idmap_addr = virt_to_phys(__hyp_idmap_text_start);
/*
* Activate the lower HYP offset only if:
* - the idmap doesn't clash with it,
* - the kernel is not running at EL2.
*/
return idmap_addr > GENMASK(VA_BITS - 2, 0) && !is_kernel_in_hyp_mode();
}
static const struct arm64_cpu_capabilities arm64_features[] = {
{
.desc = "GIC system register CPU interface",
@ -803,6 +816,12 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.field_pos = ID_AA64PFR0_EL0_SHIFT,
.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
},
{
.desc = "Reduced HYP mapping offset",
.capability = ARM64_HYP_OFFSET_LOW,
.def_scope = SCOPE_SYSTEM,
.matches = hyp_offset_low,
},
{},
};

8
arch/arm64/kvm/Kconfig
View File

@ -36,6 +36,7 @@ config KVM
select HAVE_KVM_IRQFD
select KVM_ARM_VGIC_V3
select KVM_ARM_PMU if HW_PERF_EVENTS
select HAVE_KVM_MSI
---help---
Support hosting virtualized guest machines.
We don't support KVM with 16K page tables yet, due to the multiple
@ -54,13 +55,6 @@ config KVM_ARM_PMU
Adds support for a virtual Performance Monitoring Unit (PMU) in
virtual machines.
config KVM_NEW_VGIC
bool "New VGIC implementation"
depends on KVM
default y
---help---
uses the new VGIC implementation
source drivers/vhost/Kconfig
endif # VIRTUALIZATION

9
arch/arm64/kvm/Makefile
View File

@ -20,7 +20,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += emulate.o inject_fault.o regmap.o
kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o
kvm-$(CONFIG_KVM_ARM_HOST) += guest.o debug.o reset.o sys_regs.o sys_regs_generic_v8.o
ifeq ($(CONFIG_KVM_NEW_VGIC),y)
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-init.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-irqfd.o
@ -30,12 +29,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
else
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2-emul.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v3.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v3-emul.o
endif
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o

2
arch/arm64/kvm/guest.c
View File

@ -211,7 +211,7 @@ unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
/**
* kvm_arm_copy_reg_indices - get indices of all registers.
*
* We do core registers right here, then we apppend system regs.
* We do core registers right here, then we append system regs.
*/
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
{

61
arch/arm64/kvm/hyp-init.S
View File

@ -53,10 +53,9 @@ __invalid:
b .
/*
* x0: HYP boot pgd
* x1: HYP pgd
* x2: HYP stack
* x3: HYP vectors
* x0: HYP pgd
* x1: HYP stack
* x2: HYP vectors
*/
__do_hyp_init:
@ -110,71 +109,27 @@ __do_hyp_init:
msr sctlr_el2, x4
isb
/* Skip the trampoline dance if we merged the boot and runtime PGDs */
cmp x0, x1
b.eq merged
/* MMU is now enabled. Get ready for the trampoline dance */
ldr x4, =TRAMPOLINE_VA
adr x5, target
bfi x4, x5, #0, #PAGE_SHIFT
br x4
target: /* We're now in the trampoline code, switch page tables */
msr ttbr0_el2, x1
isb
/* Invalidate the old TLBs */
tlbi alle2
dsb sy
merged:
/* Set the stack and new vectors */
kern_hyp_va x1
mov sp, x1
kern_hyp_va x2
mov sp, x2
kern_hyp_va x3
msr vbar_el2, x3
msr vbar_el2, x2
/* Hello, World! */
eret
ENDPROC(__kvm_hyp_init)
/*
* Reset kvm back to the hyp stub. This is the trampoline dance in
* reverse. If kvm used an extended idmap, __extended_idmap_trampoline
* calls this code directly in the idmap. In this case switching to the
* boot tables is a no-op.
*
* x0: HYP boot pgd
* x1: HYP phys_idmap_start
* Reset kvm back to the hyp stub.
*/
ENTRY(__kvm_hyp_reset)
/* We're in trampoline code in VA, switch back to boot page tables */
msr ttbr0_el2, x0
isb
/* Ensure the PA branch doesn't find a stale tlb entry or stale code. */
ic iallu
tlbi alle2
dsb sy
isb
/* Branch into PA space */
adr x0, 1f
bfi x1, x0, #0, #PAGE_SHIFT
br x1
/* We're now in idmap, disable MMU */
1: mrs x0, sctlr_el2
mrs x0, sctlr_el2
ldr x1, =SCTLR_ELx_FLAGS
bic x0, x0, x1 // Clear SCTL_M and etc
msr sctlr_el2, x0
isb
/* Invalidate the old TLBs */
tlbi alle2
dsb sy
/* Install stub vectors */
adr_l x0, __hyp_stub_vectors
msr vbar_el2, x0

19
arch/arm64/kvm/hyp/entry.S
View File

@ -164,22 +164,3 @@ alternative_endif
eret
ENDPROC(__fpsimd_guest_restore)
/*
* When using the extended idmap, we don't have a trampoline page we can use
* while we switch pages tables during __kvm_hyp_reset. Accessing the idmap
* directly would be ideal, but if we're using the extended idmap then the
* idmap is located above HYP_PAGE_OFFSET, and the address will be masked by
* kvm_call_hyp using kern_hyp_va.
*
* x0: HYP boot pgd
* x1: HYP phys_idmap_start
*/
ENTRY(__extended_idmap_trampoline)
mov x4, x1
adr_l x3, __kvm_hyp_reset
/* insert __kvm_hyp_reset()s offset into phys_idmap_start */
bfi x4, x3, #0, #PAGE_SHIFT
br x4
ENDPROC(__extended_idmap_trampoline)

15
arch/arm64/kvm/hyp/hyp-entry.S
View File

@ -62,6 +62,21 @@ ENTRY(__vhe_hyp_call)
isb
ret
ENDPROC(__vhe_hyp_call)
/*
* Compute the idmap address of __kvm_hyp_reset based on the idmap
* start passed as a parameter, and jump there.
*
* x0: HYP phys_idmap_start
*/
ENTRY(__kvm_hyp_teardown)
mov x4, x0
adr_l x3, __kvm_hyp_reset
/* insert __kvm_hyp_reset()s offset into phys_idmap_start */
bfi x4, x3, #0, #PAGE_SHIFT
br x4
ENDPROC(__kvm_hyp_teardown)
el1_sync: // Guest trapped into EL2
save_x0_to_x3

11
arch/arm64/kvm/hyp/switch.c
View File

@ -299,9 +299,16 @@ static const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%
static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par)
{
unsigned long str_va = (unsigned long)__hyp_panic_string;
unsigned long str_va;
__hyp_do_panic(hyp_kern_va(str_va),
/*
* Force the panic string to be loaded from the literal pool,
* making sure it is a kernel address and not a PC-relative
* reference.
*/
asm volatile("ldr %0, =__hyp_panic_string" : "=r" (str_va));
__hyp_do_panic(str_va,
spsr, elr,
read_sysreg(esr_el2), read_sysreg_el2(far),
read_sysreg(hpfar_el2), par,

38
arch/arm64/kvm/reset.c
View File

@ -65,7 +65,7 @@ static bool cpu_has_32bit_el1(void)
* We currently assume that the number of HW registers is uniform
* across all CPUs (see cpuinfo_sanity_check).
*/
int kvm_arch_dev_ioctl_check_extension(long ext)
int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
{
int r;
@ -86,6 +86,12 @@ int kvm_arch_dev_ioctl_check_extension(long ext)
case KVM_CAP_VCPU_ATTRIBUTES:
r = 1;
break;
case KVM_CAP_MSI_DEVID:
if (!kvm)
r = -EINVAL;
else
r = kvm->arch.vgic.msis_require_devid;
break;
default:
r = 0;
}
@ -98,7 +104,7 @@ int kvm_arch_dev_ioctl_check_extension(long ext)
* @vcpu: The VCPU pointer
*
* This function finds the right table above and sets the registers on
* the virtual CPU struct to their architectually defined reset
* the virtual CPU struct to their architecturally defined reset
* values.
*/
int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
@ -132,31 +138,3 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
/* Reset timer */
return kvm_timer_vcpu_reset(vcpu, cpu_vtimer_irq);
}
extern char __hyp_idmap_text_start[];
unsigned long kvm_hyp_reset_entry(void)
{
if (!__kvm_cpu_uses_extended_idmap()) {
unsigned long offset;
/*
* Find the address of __kvm_hyp_reset() in the trampoline page.
* This is present in the running page tables, and the boot page
* tables, so we call the code here to start the trampoline
* dance in reverse.
*/
offset = (unsigned long)__kvm_hyp_reset
- ((unsigned long)__hyp_idmap_text_start & PAGE_MASK);
return TRAMPOLINE_VA + offset;
} else {
/*
* KVM is running with merged page tables, which don't have the
* trampoline page mapped. We know the idmap is still mapped,
* but can't be called into directly. Use
* __extended_idmap_trampoline to do the call.
*/
return (unsigned long)kvm_ksym_ref(__extended_idmap_trampoline);
}
}

4
arch/arm64/kvm/sys_regs.c
View File

@ -1546,7 +1546,7 @@ static void unhandled_cp_access(struct kvm_vcpu *vcpu,
struct sys_reg_params *params)
{
u8 hsr_ec = kvm_vcpu_trap_get_class(vcpu);
int cp;
int cp = -1;
switch(hsr_ec) {
case ESR_ELx_EC_CP15_32:
@ -1558,7 +1558,7 @@ static void unhandled_cp_access(struct kvm_vcpu *vcpu,
cp = 14;
break;
default:
WARN_ON((cp = -1));
WARN_ON(1);
}
kvm_err("Unsupported guest CP%d access at: %08lx\n",

2
arch/mips/Kconfig
View File

@ -1488,6 +1488,7 @@ config CPU_MIPS64_R2
select CPU_SUPPORTS_HIGHMEM
select CPU_SUPPORTS_HUGEPAGES
select CPU_SUPPORTS_MSA
select HAVE_KVM
help
Choose this option to build a kernel for release 2 or later of the
MIPS64 architecture. Many modern embedded systems with a 64-bit
@ -1505,6 +1506,7 @@ config CPU_MIPS64_R6
select CPU_SUPPORTS_MSA
select GENERIC_CSUM
select MIPS_O32_FP64_SUPPORT if MIPS32_O32
select HAVE_KVM
help
Choose this option to build a kernel for release 6 or later of the
MIPS64 architecture. New MIPS processors, starting with the Warrior

2
arch/mips/include/asm/addrspace.h
View File

@ -45,7 +45,7 @@
/*
* Returns the kernel segment base of a given address
*/
#define KSEGX(a) ((_ACAST32_ (a)) & 0xe0000000)
#define KSEGX(a) ((_ACAST32_(a)) & _ACAST32_(0xe0000000))
/*
* Returns the physical address of a CKSEGx / XKPHYS address

315
arch/mips/include/asm/kvm_host.h
View File

@ -19,6 +19,9 @@
#include <linux/threads.h>
#include <linux/spinlock.h>
#include <asm/inst.h>
#include <asm/mipsregs.h>
/* MIPS KVM register ids */
#define MIPS_CP0_32(_R, _S) \
(KVM_REG_MIPS_CP0 | KVM_REG_SIZE_U32 | (8 * (_R) + (_S)))
@ -53,6 +56,12 @@
#define KVM_REG_MIPS_CP0_CONFIG7 MIPS_CP0_32(16, 7)
#define KVM_REG_MIPS_CP0_XCONTEXT MIPS_CP0_64(20, 0)
#define KVM_REG_MIPS_CP0_ERROREPC MIPS_CP0_64(30, 0)
#define KVM_REG_MIPS_CP0_KSCRATCH1 MIPS_CP0_64(31, 2)
#define KVM_REG_MIPS_CP0_KSCRATCH2 MIPS_CP0_64(31, 3)
#define KVM_REG_MIPS_CP0_KSCRATCH3 MIPS_CP0_64(31, 4)
#define KVM_REG_MIPS_CP0_KSCRATCH4 MIPS_CP0_64(31, 5)
#define KVM_REG_MIPS_CP0_KSCRATCH5 MIPS_CP0_64(31, 6)
#define KVM_REG_MIPS_CP0_KSCRATCH6 MIPS_CP0_64(31, 7)
#define KVM_MAX_VCPUS 1
@ -65,8 +74,14 @@
/* Special address that contains the comm page, used for reducing # of traps */
#define KVM_GUEST_COMMPAGE_ADDR 0x0
/*
* Special address that contains the comm page, used for reducing # of traps
* This needs to be within 32Kb of 0x0 (so the zero register can be used), but
* preferably not at 0x0 so that most kernel NULL pointer dereferences can be
* caught.
*/
#define KVM_GUEST_COMMPAGE_ADDR ((PAGE_SIZE > 0x8000) ? 0 : \
(0x8000 - PAGE_SIZE))
#define KVM_GUEST_KERNEL_MODE(vcpu) ((kvm_read_c0_guest_status(vcpu->arch.cop0) & (ST0_EXL | ST0_ERL)) || \
((kvm_read_c0_guest_status(vcpu->arch.cop0) & KSU_USER) == 0))
@ -93,9 +108,6 @@
#define KVM_INVALID_ADDR 0xdeadbeef
extern atomic_t kvm_mips_instance;
extern kvm_pfn_t (*kvm_mips_gfn_to_pfn)(struct kvm *kvm, gfn_t gfn);
extern void (*kvm_mips_release_pfn_clean)(kvm_pfn_t pfn);
extern bool (*kvm_mips_is_error_pfn)(kvm_pfn_t pfn);
struct kvm_vm_stat {
u32 remote_tlb_flush;
@ -126,28 +138,6 @@ struct kvm_vcpu_stat {
u32 halt_wakeup;
};
enum kvm_mips_exit_types {
WAIT_EXITS,
CACHE_EXITS,
SIGNAL_EXITS,
INT_EXITS,
COP_UNUSABLE_EXITS,
TLBMOD_EXITS,
TLBMISS_LD_EXITS,
TLBMISS_ST_EXITS,
ADDRERR_ST_EXITS,
ADDRERR_LD_EXITS,
SYSCALL_EXITS,
RESVD_INST_EXITS,
BREAK_INST_EXITS,
TRAP_INST_EXITS,
MSA_FPE_EXITS,
FPE_EXITS,
MSA_DISABLED_EXITS,
FLUSH_DCACHE_EXITS,
MAX_KVM_MIPS_EXIT_TYPES
};
struct kvm_arch_memory_slot {
};
@ -215,73 +205,6 @@ struct mips_coproc {
#define MIPS_CP0_CONFIG4_SEL 4
#define MIPS_CP0_CONFIG5_SEL 5
/* Config0 register bits */
#define CP0C0_M 31
#define CP0C0_K23 28
#define CP0C0_KU 25
#define CP0C0_MDU 20
#define CP0C0_MM 17
#define CP0C0_BM 16
#define CP0C0_BE 15
#define CP0C0_AT 13
#define CP0C0_AR 10
#define CP0C0_MT 7
#define CP0C0_VI 3
#define CP0C0_K0 0
/* Config1 register bits */
#define CP0C1_M 31
#define CP0C1_MMU 25
#define CP0C1_IS 22
#define CP0C1_IL 19
#define CP0C1_IA 16
#define CP0C1_DS 13
#define CP0C1_DL 10
#define CP0C1_DA 7
#define CP0C1_C2 6
#define CP0C1_MD 5
#define CP0C1_PC 4
#define CP0C1_WR 3
#define CP0C1_CA 2
#define CP0C1_EP 1
#define CP0C1_FP 0
/* Config2 Register bits */
#define CP0C2_M 31
#define CP0C2_TU 28
#define CP0C2_TS 24
#define CP0C2_TL 20
#define CP0C2_TA 16
#define CP0C2_SU 12
#define CP0C2_SS 8
#define CP0C2_SL 4
#define CP0C2_SA 0
/* Config3 Register bits */
#define CP0C3_M 31
#define CP0C3_ISA_ON_EXC 16
#define CP0C3_ULRI 13
#define CP0C3_DSPP 10
#define CP0C3_LPA 7
#define CP0C3_VEIC 6
#define CP0C3_VInt 5
#define CP0C3_SP 4
#define CP0C3_MT 2
#define CP0C3_SM 1
#define CP0C3_TL 0
/* MMU types, the first four entries have the same layout as the
CP0C0_MT field. */
enum mips_mmu_types {
MMU_TYPE_NONE,
MMU_TYPE_R4000,
MMU_TYPE_RESERVED,
MMU_TYPE_FMT,
MMU_TYPE_R3000,
MMU_TYPE_R6000,
MMU_TYPE_R8000
};
/* Resume Flags */
#define RESUME_FLAG_DR (1<<0) /* Reload guest nonvolatile state? */
#define RESUME_FLAG_HOST (1<<1) /* Resume host? */
@ -298,11 +221,6 @@ enum emulation_result {
EMULATE_PRIV_FAIL,
};
#define MIPS3_PG_G 0x00000001 /* Global; ignore ASID if in lo0 & lo1 */
#define MIPS3_PG_V 0x00000002 /* Valid */
#define MIPS3_PG_NV 0x00000000
#define MIPS3_PG_D 0x00000004 /* Dirty */
#define mips3_paddr_to_tlbpfn(x) \
(((unsigned long)(x) >> MIPS3_PG_SHIFT) & MIPS3_PG_FRAME)
#define mips3_tlbpfn_to_paddr(x) \
@ -313,13 +231,11 @@ enum emulation_result {
#define VPN2_MASK 0xffffe000
#define KVM_ENTRYHI_ASID MIPS_ENTRYHI_ASID
#define TLB_IS_GLOBAL(x) (((x).tlb_lo0 & MIPS3_PG_G) && \
((x).tlb_lo1 & MIPS3_PG_G))
#define TLB_IS_GLOBAL(x) ((x).tlb_lo[0] & (x).tlb_lo[1] & ENTRYLO_G)
#define TLB_VPN2(x) ((x).tlb_hi & VPN2_MASK)
#define TLB_ASID(x) ((x).tlb_hi & KVM_ENTRYHI_ASID)
#define TLB_IS_VALID(x, va) (((va) & (1 << PAGE_SHIFT)) \
? ((x).tlb_lo1 & MIPS3_PG_V) \
: ((x).tlb_lo0 & MIPS3_PG_V))
#define TLB_LO_IDX(x, va) (((va) >> PAGE_SHIFT) & 1)
#define TLB_IS_VALID(x, va) ((x).tlb_lo[TLB_LO_IDX(x, va)] & ENTRYLO_V)
#define TLB_HI_VPN2_HIT(x, y) ((TLB_VPN2(x) & ~(x).tlb_mask) == \
((y) & VPN2_MASK & ~(x).tlb_mask))
#define TLB_HI_ASID_HIT(x, y) (TLB_IS_GLOBAL(x) || \
@ -328,26 +244,23 @@ enum emulation_result {
struct kvm_mips_tlb {
long tlb_mask;
long tlb_hi;
long tlb_lo0;
long tlb_lo1;
long tlb_lo[2];
};
#define KVM_MIPS_FPU_FPU 0x1
#define KVM_MIPS_FPU_MSA 0x2
#define KVM_MIPS_AUX_FPU 0x1
#define KVM_MIPS_AUX_MSA 0x2
#define KVM_MIPS_GUEST_TLB_SIZE 64
struct kvm_vcpu_arch {
void *host_ebase, *guest_ebase;
void *guest_ebase;
int (*vcpu_run)(struct kvm_run *run, struct kvm_vcpu *vcpu);
unsigned long host_stack;
unsigned long host_gp;
/* Host CP0 registers used when handling exits from guest */
unsigned long host_cp0_badvaddr;
unsigned long host_cp0_cause;
unsigned long host_cp0_epc;
unsigned long host_cp0_entryhi;
uint32_t guest_inst;
u32 host_cp0_cause;
/* GPRS */
unsigned long gprs[32];
@ -357,8 +270,8 @@ struct kvm_vcpu_arch {
/* FPU State */
struct mips_fpu_struct fpu;
/* Which FPU state is loaded (KVM_MIPS_FPU_*) */
unsigned int fpu_inuse;
/* Which auxiliary state is loaded (KVM_MIPS_AUX_*) */
unsigned int aux_inuse;
/* COP0 State */
struct mips_coproc *cop0;
@ -370,11 +283,11 @@ struct kvm_vcpu_arch {
struct hrtimer comparecount_timer;
/* Count timer control KVM register */
uint32_t count_ctl;
u32 count_ctl;
/* Count bias from the raw time */
uint32_t count_bias;
u32 count_bias;
/* Frequency of timer in Hz */
uint32_t count_hz;
u32 count_hz;
/* Dynamic nanosecond bias (multiple of count_period) to avoid overflow */
s64 count_dyn_bias;
/* Resume time */
@ -388,7 +301,7 @@ struct kvm_vcpu_arch {
/* Bitmask of pending exceptions to be cleared */
unsigned long pending_exceptions_clr;
unsigned long pending_load_cause;
u32 pending_load_cause;
/* Save/Restore the entryhi register when are are preempted/scheduled back in */
unsigned long preempt_entryhi;
@ -397,8 +310,8 @@ struct kvm_vcpu_arch {
struct kvm_mips_tlb guest_tlb[KVM_MIPS_GUEST_TLB_SIZE];
/* Cached guest kernel/user ASIDs */
uint32_t guest_user_asid[NR_CPUS];
uint32_t guest_kernel_asid[NR_CPUS];
u32 guest_user_asid[NR_CPUS];
u32 guest_kernel_asid[NR_CPUS];
struct mm_struct guest_kernel_mm, guest_user_mm;
int last_sched_cpu;
@ -408,6 +321,7 @@ struct kvm_vcpu_arch {
u8 fpu_enabled;
u8 msa_enabled;
u8 kscratch_enabled;
};
@ -461,6 +375,18 @@ struct kvm_vcpu_arch {
#define kvm_write_c0_guest_config7(cop0, val) (cop0->reg[MIPS_CP0_CONFIG][7] = (val))
#define kvm_read_c0_guest_errorepc(cop0) (cop0->reg[MIPS_CP0_ERROR_PC][0])
#define kvm_write_c0_guest_errorepc(cop0, val) (cop0->reg[MIPS_CP0_ERROR_PC][0] = (val))
#define kvm_read_c0_guest_kscratch1(cop0) (cop0->reg[MIPS_CP0_DESAVE][2])
#define kvm_read_c0_guest_kscratch2(cop0) (cop0->reg[MIPS_CP0_DESAVE][3])
#define kvm_read_c0_guest_kscratch3(cop0) (cop0->reg[MIPS_CP0_DESAVE][4])
#define kvm_read_c0_guest_kscratch4(cop0) (cop0->reg[MIPS_CP0_DESAVE][5])
#define kvm_read_c0_guest_kscratch5(cop0) (cop0->reg[MIPS_CP0_DESAVE][6])
#define kvm_read_c0_guest_kscratch6(cop0) (cop0->reg[MIPS_CP0_DESAVE][7])
#define kvm_write_c0_guest_kscratch1(cop0, val) (cop0->reg[MIPS_CP0_DESAVE][2] = (val))
#define kvm_write_c0_guest_kscratch2(cop0, val) (cop0->reg[MIPS_CP0_DESAVE][3] = (val))
#define kvm_write_c0_guest_kscratch3(cop0, val) (cop0->reg[MIPS_CP0_DESAVE][4] = (val))
#define kvm_write_c0_guest_kscratch4(cop0, val) (cop0->reg[MIPS_CP0_DESAVE][5] = (val))
#define kvm_write_c0_guest_kscratch5(cop0, val) (cop0->reg[MIPS_CP0_DESAVE][6] = (val))
#define kvm_write_c0_guest_kscratch6(cop0, val) (cop0->reg[MIPS_CP0_DESAVE][7] = (val))
/*
* Some of the guest registers may be modified asynchronously (e.g. from a
@ -474,7 +400,7 @@ static inline void _kvm_atomic_set_c0_guest_reg(unsigned long *reg,
unsigned long temp;
do {
__asm__ __volatile__(
" .set mips3 \n"
" .set "MIPS_ISA_ARCH_LEVEL" \n"
" " __LL "%0, %1 \n"
" or %0, %2 \n"
" " __SC "%0, %1 \n"
@ -490,7 +416,7 @@ static inline void _kvm_atomic_clear_c0_guest_reg(unsigned long *reg,
unsigned long temp;
do {
__asm__ __volatile__(
" .set mips3 \n"
" .set "MIPS_ISA_ARCH_LEVEL" \n"
" " __LL "%0, %1 \n"
" and %0, %2 \n"
" " __SC "%0, %1 \n"
@ -507,7 +433,7 @@ static inline void _kvm_atomic_change_c0_guest_reg(unsigned long *reg,
unsigned long temp;
do {
__asm__ __volatile__(
" .set mips3 \n"
" .set "MIPS_ISA_ARCH_LEVEL" \n"
" " __LL "%0, %1 \n"
" and %0, %2 \n"
" or %0, %3 \n"
@ -542,7 +468,7 @@ static inline void _kvm_atomic_change_c0_guest_reg(unsigned long *reg,
static inline bool kvm_mips_guest_can_have_fpu(struct kvm_vcpu_arch *vcpu)
{
return (!__builtin_constant_p(cpu_has_fpu) || cpu_has_fpu) &&
return (!__builtin_constant_p(raw_cpu_has_fpu) || raw_cpu_has_fpu) &&
vcpu->fpu_enabled;
}
@ -589,9 +515,11 @@ struct kvm_mips_callbacks {
void (*dequeue_io_int)(struct kvm_vcpu *vcpu,
struct kvm_mips_interrupt *irq);
int (*irq_deliver)(struct kvm_vcpu *vcpu, unsigned int priority,
uint32_t cause);
u32 cause);
int (*irq_clear)(struct kvm_vcpu *vcpu, unsigned int priority,
uint32_t cause);
u32 cause);
unsigned long (*num_regs)(struct kvm_vcpu *vcpu);
int (*copy_reg_indices)(struct kvm_vcpu *vcpu, u64 __user *indices);
int (*get_one_reg)(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg, s64 *v);
int (*set_one_reg)(struct kvm_vcpu *vcpu,
@ -605,8 +533,13 @@ int kvm_mips_emulation_init(struct kvm_mips_callbacks **install_callbacks);
/* Debug: dump vcpu state */
int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu);
/* Trampoline ASM routine to start running in "Guest" context */
extern int __kvm_mips_vcpu_run(struct kvm_run *run, struct kvm_vcpu *vcpu);
extern int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu);
/* Building of entry/exception code */
int kvm_mips_entry_setup(void);
void *kvm_mips_build_vcpu_run(void *addr);
void *kvm_mips_build_exception(void *addr, void *handler);
void *kvm_mips_build_exit(void *addr);
/* FPU/MSA context management */
void __kvm_save_fpu(struct kvm_vcpu_arch *vcpu);
@ -622,11 +555,11 @@ void kvm_drop_fpu(struct kvm_vcpu *vcpu);
void kvm_lose_fpu(struct kvm_vcpu *vcpu);
/* TLB handling */
uint32_t kvm_get_kernel_asid(struct kvm_vcpu *vcpu);
u32 kvm_get_kernel_asid(struct kvm_vcpu *vcpu);
uint32_t kvm_get_user_asid(struct kvm_vcpu *vcpu);
u32 kvm_get_user_asid(struct kvm_vcpu *vcpu);
uint32_t kvm_get_commpage_asid (struct kvm_vcpu *vcpu);
u32 kvm_get_commpage_asid (struct kvm_vcpu *vcpu);
extern int kvm_mips_handle_kseg0_tlb_fault(unsigned long badbaddr,
struct kvm_vcpu *vcpu);
@ -635,22 +568,24 @@ extern int kvm_mips_handle_commpage_tlb_fault(unsigned long badvaddr,
struct kvm_vcpu *vcpu);
extern int kvm_mips_handle_mapped_seg_tlb_fault(struct kvm_vcpu *vcpu,
struct kvm_mips_tlb *tlb,
unsigned long *hpa0,
unsigned long *hpa1);
struct kvm_mips_tlb *tlb);
extern enum emulation_result kvm_mips_handle_tlbmiss(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_handle_tlbmiss(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_handle_tlbmod(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_handle_tlbmod(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern void kvm_mips_dump_host_tlbs(void);
extern void kvm_mips_dump_guest_tlbs(struct kvm_vcpu *vcpu);
extern int kvm_mips_host_tlb_write(struct kvm_vcpu *vcpu, unsigned long entryhi,
unsigned long entrylo0,
unsigned long entrylo1,
int flush_dcache_mask);
extern void kvm_mips_flush_host_tlb(int skip_kseg0);
extern int kvm_mips_host_tlb_inv(struct kvm_vcpu *vcpu, unsigned long entryhi);
@ -667,90 +602,90 @@ extern void kvm_mips_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
extern void kvm_mips_vcpu_put(struct kvm_vcpu *vcpu);
/* Emulation */
uint32_t kvm_get_inst(uint32_t *opc, struct kvm_vcpu *vcpu);
enum emulation_result update_pc(struct kvm_vcpu *vcpu, uint32_t cause);
u32 kvm_get_inst(u32 *opc, struct kvm_vcpu *vcpu);
enum emulation_result update_pc(struct kvm_vcpu *vcpu, u32 cause);
extern enum emulation_result kvm_mips_emulate_inst(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_inst(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_syscall(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_syscall(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_tlbmiss_ld(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_tlbmiss_ld(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_tlbinv_ld(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_tlbinv_ld(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_tlbmiss_st(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_tlbmiss_st(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_tlbinv_st(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_tlbinv_st(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_tlbmod(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_tlbmod(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_fpu_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_fpu_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_handle_ri(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_handle_ri(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_ri_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_ri_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_bp_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_bp_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_trap_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_trap_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_msafpe_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_msafpe_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_fpe_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_fpe_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_emulate_msadis_exc(unsigned long cause,
uint32_t *opc,
extern enum emulation_result kvm_mips_emulate_msadis_exc(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
extern enum emulation_result kvm_mips_complete_mmio_load(struct kvm_vcpu *vcpu,
struct kvm_run *run);
uint32_t kvm_mips_read_count(struct kvm_vcpu *vcpu);
void kvm_mips_write_count(struct kvm_vcpu *vcpu, uint32_t count);
void kvm_mips_write_compare(struct kvm_vcpu *vcpu, uint32_t compare, bool ack);
u32 kvm_mips_read_count(struct kvm_vcpu *vcpu);
void kvm_mips_write_count(struct kvm_vcpu *vcpu, u32 count);
void kvm_mips_write_compare(struct kvm_vcpu *vcpu, u32 compare, bool ack);
void kvm_mips_init_count(struct kvm_vcpu *vcpu);
int kvm_mips_set_count_ctl(struct kvm_vcpu *vcpu, s64 count_ctl);
int kvm_mips_set_count_resume(struct kvm_vcpu *vcpu, s64 count_resume);
@ -759,27 +694,27 @@ void kvm_mips_count_enable_cause(struct kvm_vcpu *vcpu);
void kvm_mips_count_disable_cause(struct kvm_vcpu *vcpu);
enum hrtimer_restart kvm_mips_count_timeout(struct kvm_vcpu *vcpu);
enum emulation_result kvm_mips_check_privilege(unsigned long cause,
uint32_t *opc,
enum emulation_result kvm_mips_check_privilege(u32 cause,
u32 *opc,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
enum emulation_result kvm_mips_emulate_cache(uint32_t inst,
uint32_t *opc,
uint32_t cause,
enum emulation_result kvm_mips_emulate_cache(union mips_instruction inst,
u32 *opc,
u32 cause,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
enum emulation_result kvm_mips_emulate_CP0(uint32_t inst,
uint32_t *opc,
uint32_t cause,
enum emulation_result kvm_mips_emulate_CP0(union mips_instruction inst,
u32 *opc,
u32 cause,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
enum emulation_result kvm_mips_emulate_store(uint32_t inst,
uint32_t cause,
enum emulation_result kvm_mips_emulate_store(union mips_instruction inst,
u32 cause,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
enum emulation_result kvm_mips_emulate_load(uint32_t inst,
uint32_t cause,
enum emulation_result kvm_mips_emulate_load(union mips_instruction inst,
u32 cause,
struct kvm_run *run,
struct kvm_vcpu *vcpu);
@ -789,13 +724,13 @@ unsigned int kvm_mips_config4_wrmask(struct kvm_vcpu *vcpu);
unsigned int kvm_mips_config5_wrmask(struct kvm_vcpu *vcpu);
/* Dynamic binary translation */
extern int kvm_mips_trans_cache_index(uint32_t inst, uint32_t *opc,
struct kvm_vcpu *vcpu);
extern int kvm_mips_trans_cache_va(uint32_t inst, uint32_t *opc,
extern int kvm_mips_trans_cache_index(union mips_instruction inst,
u32 *opc, struct kvm_vcpu *vcpu);
extern int kvm_mips_trans_cache_va(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu);
extern int kvm_mips_trans_mfc0(uint32_t inst, uint32_t *opc,
extern int kvm_mips_trans_mfc0(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu);
extern int kvm_mips_trans_mtc0(uint32_t inst, uint32_t *opc,
extern int kvm_mips_trans_mtc0(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu);
/* Misc */

2
arch/mips/include/asm/mach-cavium-octeon/cpu-feature-overrides.h
View File

@ -55,7 +55,7 @@
#define cpu_has_mipsmt 0
#define cpu_has_vint 0
#define cpu_has_veic 0
#define cpu_hwrena_impl_bits 0xc0000000
#define cpu_hwrena_impl_bits (MIPS_HWRENA_IMPL1 | MIPS_HWRENA_IMPL2)
#define cpu_has_wsbh 1
#define cpu_has_rixi (cpu_data[0].cputype != CPU_CAVIUM_OCTEON)

21
arch/mips/include/asm/mipsregs.h
View File

@ -53,7 +53,7 @@
#define CP0_SEGCTL2 $5, 4
#define CP0_WIRED $6
#define CP0_INFO $7
#define CP0_HWRENA $7, 0
#define CP0_HWRENA $7
#define CP0_BADVADDR $8
#define CP0_BADINSTR $8, 1
#define CP0_COUNT $9
@ -533,6 +533,7 @@
#define TX49_CONF_CWFON (_ULCAST_(1) << 27)
/* Bits specific to the MIPS32/64 PRA. */
#define MIPS_CONF_VI (_ULCAST_(1) << 3)
#define MIPS_CONF_MT (_ULCAST_(7) << 7)
#define MIPS_CONF_MT_TLB (_ULCAST_(1) << 7)
#define MIPS_CONF_MT_FTLB (_ULCAST_(4) << 7)
@ -853,6 +854,24 @@
#define MIPS_CDMMBASE_ADDR_SHIFT 11
#define MIPS_CDMMBASE_ADDR_START 15
/* RDHWR register numbers */
#define MIPS_HWR_CPUNUM 0 /* CPU number */
#define MIPS_HWR_SYNCISTEP 1 /* SYNCI step size */
#define MIPS_HWR_CC 2 /* Cycle counter */
#define MIPS_HWR_CCRES 3 /* Cycle counter resolution */
#define MIPS_HWR_ULR 29 /* UserLocal */
#define MIPS_HWR_IMPL1 30 /* Implementation dependent */
#define MIPS_HWR_IMPL2 31 /* Implementation dependent */
/* Bits in HWREna register */
#define MIPS_HWRENA_CPUNUM (_ULCAST_(1) << MIPS_HWR_CPUNUM)
#define MIPS_HWRENA_SYNCISTEP (_ULCAST_(1) << MIPS_HWR_SYNCISTEP)
#define MIPS_HWRENA_CC (_ULCAST_(1) << MIPS_HWR_CC)
#define MIPS_HWRENA_CCRES (_ULCAST_(1) << MIPS_HWR_CCRES)
#define MIPS_HWRENA_ULR (_ULCAST_(1) << MIPS_HWR_ULR)
#define MIPS_HWRENA_IMPL1 (_ULCAST_(1) << MIPS_HWR_IMPL1)
#define MIPS_HWRENA_IMPL2 (_ULCAST_(1) << MIPS_HWR_IMPL2)
/*
* Bitfields in the TX39 family CP0 Configuration Register 3
*/

1
arch/mips/include/asm/setup.h
View File

@ -21,6 +21,7 @@ extern void *set_vi_handler(int n, vi_handler_t addr);
extern void *set_except_vector(int n, void *addr);
extern unsigned long ebase;
extern unsigned int hwrena;
extern void per_cpu_trap_init(bool);
extern void cpu_cache_init(void);

7
arch/mips/include/asm/uasm.h
View File

@ -104,8 +104,13 @@ Ip_u1s2(_bltz);
Ip_u1s2(_bltzl);
Ip_u1u2s3(_bne);
Ip_u2s3u1(_cache);
Ip_u1u2(_cfc1);
Ip_u2u1(_cfcmsa);
Ip_u1u2(_ctc1);
Ip_u2u1(_ctcmsa);
Ip_u2u1s3(_daddiu);
Ip_u3u1u2(_daddu);
Ip_u1(_di);
Ip_u2u1msbu3(_dins);
Ip_u2u1msbu3(_dinsm);
Ip_u1u2(_divu);
@ -141,6 +146,8 @@ Ip_u1(_mfhi);
Ip_u1(_mflo);
Ip_u1u2u3(_mtc0);
Ip_u1u2u3(_mthc0);
Ip_u1(_mthi);
Ip_u1(_mtlo);
Ip_u3u1u2(_mul);
Ip_u3u1u2(_or);
Ip_u2u1u3(_ori);

114
arch/mips/include/uapi/asm/inst.h
View File

@ -21,20 +21,20 @@
enum major_op {
spec_op, bcond_op, j_op, jal_op,
beq_op, bne_op, blez_op, bgtz_op,
addi_op, cbcond0_op = addi_op, addiu_op, slti_op, sltiu_op,
addi_op, pop10_op = addi_op, addiu_op, slti_op, sltiu_op,
andi_op, ori_op, xori_op, lui_op,
cop0_op, cop1_op, cop2_op, cop1x_op,
beql_op, bnel_op, blezl_op, bgtzl_op,
daddi_op, cbcond1_op = daddi_op, daddiu_op, ldl_op, ldr_op,
daddi_op, pop30_op = daddi_op, daddiu_op, ldl_op, ldr_op,
spec2_op, jalx_op, mdmx_op, msa_op = mdmx_op, spec3_op,
lb_op, lh_op, lwl_op, lw_op,
lbu_op, lhu_op, lwr_op, lwu_op,
sb_op, sh_op, swl_op, sw_op,
sdl_op, sdr_op, swr_op, cache_op,
ll_op, lwc1_op, lwc2_op, bc6_op = lwc2_op, pref_op,
lld_op, ldc1_op, ldc2_op, beqzcjic_op = ldc2_op, ld_op,
lld_op, ldc1_op, ldc2_op, pop66_op = ldc2_op, ld_op,
sc_op, swc1_op, swc2_op, balc6_op = swc2_op, major_3b_op,
scd_op, sdc1_op, sdc2_op, bnezcjialc_op = sdc2_op, sd_op
scd_op, sdc1_op, sdc2_op, pop76_op = sdc2_op, sd_op
};
/*
@ -92,6 +92,50 @@ enum spec3_op {
rdhwr_op = 0x3b
};
/*
* Bits 10-6 minor opcode for r6 spec mult/div encodings
*/
enum mult_op {
mult_mult_op = 0x0,
mult_mul_op = 0x2,
mult_muh_op = 0x3,
};
enum multu_op {
multu_multu_op = 0x0,
multu_mulu_op = 0x2,
multu_muhu_op = 0x3,
};
enum div_op {
div_div_op = 0x0,
div_div6_op = 0x2,
div_mod_op = 0x3,
};
enum divu_op {
divu_divu_op = 0x0,
divu_divu6_op = 0x2,
divu_modu_op = 0x3,
};
enum dmult_op {
dmult_dmult_op = 0x0,
dmult_dmul_op = 0x2,
dmult_dmuh_op = 0x3,
};
enum dmultu_op {
dmultu_dmultu_op = 0x0,
dmultu_dmulu_op = 0x2,
dmultu_dmuhu_op = 0x3,
};
enum ddiv_op {
ddiv_ddiv_op = 0x0,
ddiv_ddiv6_op = 0x2,
ddiv_dmod_op = 0x3,
};
enum ddivu_op {
ddivu_ddivu_op = 0x0,
ddivu_ddivu6_op = 0x2,
ddivu_dmodu_op = 0x3,
};
/*
* rt field of bcond opcodes.
*/
@ -103,7 +147,7 @@ enum rt_op {
bltzal_op, bgezal_op, bltzall_op, bgezall_op,
rt_op_0x14, rt_op_0x15, rt_op_0x16, rt_op_0x17,
rt_op_0x18, rt_op_0x19, rt_op_0x1a, rt_op_0x1b,
bposge32_op, rt_op_0x1d, rt_op_0x1e, rt_op_0x1f
bposge32_op, rt_op_0x1d, rt_op_0x1e, synci_op
};
/*
@ -237,6 +281,21 @@ enum bshfl_func {
seh_op = 0x18,
};
/*
* MSA minor opcodes.
*/
enum msa_func {
msa_elm_op = 0x19,
};
/*
* MSA ELM opcodes.
*/
enum msa_elm {
msa_ctc_op = 0x3e,
msa_cfc_op = 0x7e,
};
/*
* func field for MSA MI10 format.
*/
@ -264,7 +323,7 @@ enum mm_major_op {
mm_pool32b_op, mm_pool16b_op, mm_lhu16_op, mm_andi16_op,
mm_addiu32_op, mm_lhu32_op, mm_sh32_op, mm_lh32_op,
mm_pool32i_op, mm_pool16c_op, mm_lwsp16_op, mm_pool16d_op,
mm_ori32_op, mm_pool32f_op, mm_reserved1_op, mm_reserved2_op,
mm_ori32_op, mm_pool32f_op, mm_pool32s_op, mm_reserved2_op,
mm_pool32c_op, mm_lwgp16_op, mm_lw16_op, mm_pool16e_op,
mm_xori32_op, mm_jals32_op, mm_addiupc_op, mm_reserved3_op,
mm_reserved4_op, mm_pool16f_op, mm_sb16_op, mm_beqz16_op,
@ -360,7 +419,10 @@ enum mm_32axf_minor_op {
mm_mflo32_op = 0x075,
mm_jalrhb_op = 0x07c,
mm_tlbwi_op = 0x08d,
mm_mthi32_op = 0x0b5,
mm_tlbwr_op = 0x0cd,
mm_mtlo32_op = 0x0f5,
mm_di_op = 0x11d,
mm_jalrs_op = 0x13c,
mm_jalrshb_op = 0x17c,
mm_sync_op = 0x1ad,
@ -478,6 +540,13 @@ enum mm_32f_73_minor_op {
mm_fcvts1_op = 0xed,
};
/*
* (microMIPS) POOL32S minor opcodes.
*/
enum mm_32s_minor_op {
mm_32s_elm_op = 0x16,
};
/*
* (microMIPS) POOL16C minor opcodes.
*/
@ -586,6 +655,36 @@ struct r_format { /* Register format */
;))))))
};
struct c0r_format { /* C0 register format */
__BITFIELD_FIELD(unsigned int opcode : 6,
__BITFIELD_FIELD(unsigned int rs : 5,
__BITFIELD_FIELD(unsigned int rt : 5,
__BITFIELD_FIELD(unsigned int rd : 5,
__BITFIELD_FIELD(unsigned int z: 8,
__BITFIELD_FIELD(unsigned int sel : 3,
;))))))
};
struct mfmc0_format { /* MFMC0 register format */
__BITFIELD_FIELD(unsigned int opcode : 6,
__BITFIELD_FIELD(unsigned int rs : 5,
__BITFIELD_FIELD(unsigned int rt : 5,
__BITFIELD_FIELD(unsigned int rd : 5,
__BITFIELD_FIELD(unsigned int re : 5,
__BITFIELD_FIELD(unsigned int sc : 1,
__BITFIELD_FIELD(unsigned int : 2,
__BITFIELD_FIELD(unsigned int sel : 3,
;))))))))
};
struct co_format { /* C0 CO format */
__BITFIELD_FIELD(unsigned int opcode : 6,
__BITFIELD_FIELD(unsigned int co : 1,
__BITFIELD_FIELD(unsigned int code : 19,
__BITFIELD_FIELD(unsigned int func : 6,
;))))
};
struct p_format { /* Performance counter format (R10000) */
__BITFIELD_FIELD(unsigned int opcode : 6,
__BITFIELD_FIELD(unsigned int rs : 5,
@ -937,6 +1036,9 @@ union mips_instruction {
struct u_format u_format;
struct c_format c_format;
struct r_format r_format;
struct c0r_format c0r_format;
struct mfmc0_format mfmc0_format;
struct co_format co_format;
struct p_format p_format;
struct f_format f_format;
struct ma_format ma_format;

70
arch/mips/kernel/asm-offsets.c
View File

@ -339,71 +339,9 @@ void output_pm_defines(void)
}
#endif
void output_cpuinfo_defines(void)
{
COMMENT(" MIPS cpuinfo offsets. ");
DEFINE(CPUINFO_SIZE, sizeof(struct cpuinfo_mips));
#ifdef CONFIG_MIPS_ASID_BITS_VARIABLE
OFFSET(CPUINFO_ASID_MASK, cpuinfo_mips, asid_mask);
#endif
}
void output_kvm_defines(void)
{
COMMENT(" KVM/MIPS Specfic offsets. ");
DEFINE(VCPU_ARCH_SIZE, sizeof(struct kvm_vcpu_arch));
OFFSET(VCPU_RUN, kvm_vcpu, run);
OFFSET(VCPU_HOST_ARCH, kvm_vcpu, arch);
OFFSET(VCPU_HOST_EBASE, kvm_vcpu_arch, host_ebase);
OFFSET(VCPU_GUEST_EBASE, kvm_vcpu_arch, guest_ebase);
OFFSET(VCPU_HOST_STACK, kvm_vcpu_arch, host_stack);
OFFSET(VCPU_HOST_GP, kvm_vcpu_arch, host_gp);
OFFSET(VCPU_HOST_CP0_BADVADDR, kvm_vcpu_arch, host_cp0_badvaddr);
OFFSET(VCPU_HOST_CP0_CAUSE, kvm_vcpu_arch, host_cp0_cause);
OFFSET(VCPU_HOST_EPC, kvm_vcpu_arch, host_cp0_epc);
OFFSET(VCPU_HOST_ENTRYHI, kvm_vcpu_arch, host_cp0_entryhi);
OFFSET(VCPU_GUEST_INST, kvm_vcpu_arch, guest_inst);
OFFSET(VCPU_R0, kvm_vcpu_arch, gprs[0]);
OFFSET(VCPU_R1, kvm_vcpu_arch, gprs[1]);
OFFSET(VCPU_R2, kvm_vcpu_arch, gprs[2]);
OFFSET(VCPU_R3, kvm_vcpu_arch, gprs[3]);
OFFSET(VCPU_R4, kvm_vcpu_arch, gprs[4]);
OFFSET(VCPU_R5, kvm_vcpu_arch, gprs[5]);
OFFSET(VCPU_R6, kvm_vcpu_arch, gprs[6]);
OFFSET(VCPU_R7, kvm_vcpu_arch, gprs[7]);
OFFSET(VCPU_R8, kvm_vcpu_arch, gprs[8]);
OFFSET(VCPU_R9, kvm_vcpu_arch, gprs[9]);
OFFSET(VCPU_R10, kvm_vcpu_arch, gprs[10]);
OFFSET(VCPU_R11, kvm_vcpu_arch, gprs[11]);
OFFSET(VCPU_R12, kvm_vcpu_arch, gprs[12]);
OFFSET(VCPU_R13, kvm_vcpu_arch, gprs[13]);
OFFSET(VCPU_R14, kvm_vcpu_arch, gprs[14]);
OFFSET(VCPU_R15, kvm_vcpu_arch, gprs[15]);
OFFSET(VCPU_R16, kvm_vcpu_arch, gprs[16]);
OFFSET(VCPU_R17, kvm_vcpu_arch, gprs[17]);
OFFSET(VCPU_R18, kvm_vcpu_arch, gprs[18]);
OFFSET(VCPU_R19, kvm_vcpu_arch, gprs[19]);
OFFSET(VCPU_R20, kvm_vcpu_arch, gprs[20]);
OFFSET(VCPU_R21, kvm_vcpu_arch, gprs[21]);
OFFSET(VCPU_R22, kvm_vcpu_arch, gprs[22]);
OFFSET(VCPU_R23, kvm_vcpu_arch, gprs[23]);
OFFSET(VCPU_R24, kvm_vcpu_arch, gprs[24]);
OFFSET(VCPU_R25, kvm_vcpu_arch, gprs[25]);
OFFSET(VCPU_R26, kvm_vcpu_arch, gprs[26]);
OFFSET(VCPU_R27, kvm_vcpu_arch, gprs[27]);
OFFSET(VCPU_R28, kvm_vcpu_arch, gprs[28]);
OFFSET(VCPU_R29, kvm_vcpu_arch, gprs[29]);
OFFSET(VCPU_R30, kvm_vcpu_arch, gprs[30]);
OFFSET(VCPU_R31, kvm_vcpu_arch, gprs[31]);
OFFSET(VCPU_LO, kvm_vcpu_arch, lo);
OFFSET(VCPU_HI, kvm_vcpu_arch, hi);
OFFSET(VCPU_PC, kvm_vcpu_arch, pc);
BLANK();
OFFSET(VCPU_FPR0, kvm_vcpu_arch, fpu.fpr[0]);
OFFSET(VCPU_FPR1, kvm_vcpu_arch, fpu.fpr[1]);
@ -441,14 +379,6 @@ void output_kvm_defines(void)
OFFSET(VCPU_FCR31, kvm_vcpu_arch, fpu.fcr31);
OFFSET(VCPU_MSA_CSR, kvm_vcpu_arch, fpu.msacsr);
BLANK();
OFFSET(VCPU_COP0, kvm_vcpu_arch, cop0);
OFFSET(VCPU_GUEST_KERNEL_ASID, kvm_vcpu_arch, guest_kernel_asid);
OFFSET(VCPU_GUEST_USER_ASID, kvm_vcpu_arch, guest_user_asid);
OFFSET(COP0_TLB_HI, mips_coproc, reg[MIPS_CP0_TLB_HI][0]);
OFFSET(COP0_STATUS, mips_coproc, reg[MIPS_CP0_STATUS][0]);
BLANK();
}
#ifdef CONFIG_MIPS_CPS

8
arch/mips/kernel/branch.c
View File

@ -790,7 +790,7 @@ int __compute_return_epc_for_insn(struct pt_regs *regs,
epc += 4 + (insn.i_format.simmediate << 2);
regs->cp0_epc = epc;
break;
case beqzcjic_op:
case pop66_op:
if (!cpu_has_mips_r6) {
ret = -SIGILL;
break;
@ -798,7 +798,7 @@ int __compute_return_epc_for_insn(struct pt_regs *regs,
/* Compact branch: BEQZC || JIC */
regs->cp0_epc += 8;
break;
case bnezcjialc_op:
case pop76_op:
if (!cpu_has_mips_r6) {
ret = -SIGILL;
break;
@ -809,8 +809,8 @@ int __compute_return_epc_for_insn(struct pt_regs *regs,
regs->cp0_epc += 8;
break;
#endif
case cbcond0_op:
case cbcond1_op:
case pop10_op:
case pop30_op:
/* Only valid for MIPS R6 */
if (!cpu_has_mips_r6) {
ret = -SIGILL;

23
arch/mips/kernel/traps.c
View File

@ -619,17 +619,17 @@ static int simulate_rdhwr(struct pt_regs *regs, int rd, int rt)
perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS,
1, regs, 0);
switch (rd) {
case 0: /* CPU number */
case MIPS_HWR_CPUNUM: /* CPU number */
regs->regs[rt] = smp_processor_id();
return 0;
case 1: /* SYNCI length */
case MIPS_HWR_SYNCISTEP: /* SYNCI length */
regs->regs[rt] = min(current_cpu_data.dcache.linesz,
current_cpu_data.icache.linesz);
return 0;
case 2: /* Read count register */
case MIPS_HWR_CC: /* Read count register */
regs->regs[rt] = read_c0_count();
return 0;
case 3: /* Count register resolution */
case MIPS_HWR_CCRES: /* Count register resolution */
switch (current_cpu_type()) {
case CPU_20KC:
case CPU_25KF:
@ -639,7 +639,7 @@ static int simulate_rdhwr(struct pt_regs *regs, int rd, int rt)
regs->regs[rt] = 2;
}
return 0;
case 29:
case MIPS_HWR_ULR: /* Read UserLocal register */
regs->regs[rt] = ti->tp_value;
return 0;
default:
@ -1859,6 +1859,7 @@ void __noreturn nmi_exception_handler(struct pt_regs *regs)
#define VECTORSPACING 0x100 /* for EI/VI mode */
unsigned long ebase;
EXPORT_SYMBOL_GPL(ebase);
unsigned long exception_handlers[32];
unsigned long vi_handlers[64];
@ -2063,16 +2064,22 @@ static void configure_status(void)
status_set);
}
unsigned int hwrena;
EXPORT_SYMBOL_GPL(hwrena);
/* configure HWRENA register */
static void configure_hwrena(void)
{
unsigned int hwrena = cpu_hwrena_impl_bits;
hwrena = cpu_hwrena_impl_bits;
if (cpu_has_mips_r2_r6)
hwrena |= 0x0000000f;
hwrena |= MIPS_HWRENA_CPUNUM |
MIPS_HWRENA_SYNCISTEP |
MIPS_HWRENA_CC |
MIPS_HWRENA_CCRES;
if (!noulri && cpu_has_userlocal)
hwrena |= (1 << 29);
hwrena |= MIPS_HWRENA_ULR;
if (hwrena)
write_c0_hwrena(hwrena);

1
arch/mips/kvm/Kconfig
View File

@ -17,6 +17,7 @@ if VIRTUALIZATION
config KVM
tristate "Kernel-based Virtual Machine (KVM) support"
depends on HAVE_KVM
select EXPORT_UASM
select PREEMPT_NOTIFIERS
select ANON_INODES
select KVM_MMIO

3
arch/mips/kvm/Makefile
View File

@ -7,9 +7,10 @@ EXTRA_CFLAGS += -Ivirt/kvm -Iarch/mips/kvm
common-objs-$(CONFIG_CPU_HAS_MSA) += msa.o
kvm-objs := $(common-objs-y) mips.o emulate.o locore.o \
kvm-objs := $(common-objs-y) mips.o emulate.o entry.o \
interrupt.o stats.o commpage.o \
dyntrans.o trap_emul.o fpu.o
kvm-objs += mmu.o
obj-$(CONFIG_KVM) += kvm.o
obj-y += callback.o tlb.o

2
arch/mips/kvm/commpage.c
View File

@ -4,7 +4,7 @@
* for more details.
*
* commpage, currently used for Virtual COP0 registers.
* Mapped into the guest kernel @ 0x0.
* Mapped into the guest kernel @ KVM_GUEST_COMMPAGE_ADDR.
*
* Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
* Authors: Sanjay Lal <sanjayl@kymasys.com>

172
arch/mips/kvm/dyntrans.c
View File

@ -11,6 +11,7 @@
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/highmem.h>
#include <linux/kvm_host.h>
#include <linux/module.h>
#include <linux/vmalloc.h>
@ -20,125 +21,114 @@
#include "commpage.h"
#define SYNCI_TEMPLATE 0x041f0000
#define SYNCI_BASE(x) (((x) >> 21) & 0x1f)
#define SYNCI_OFFSET ((x) & 0xffff)
/**
* kvm_mips_trans_replace() - Replace trapping instruction in guest memory.
* @vcpu: Virtual CPU.
* @opc: PC of instruction to replace.
* @replace: Instruction to write
*/
static int kvm_mips_trans_replace(struct kvm_vcpu *vcpu, u32 *opc,
union mips_instruction replace)
{
unsigned long paddr, flags;
void *vaddr;
#define LW_TEMPLATE 0x8c000000
#define CLEAR_TEMPLATE 0x00000020
#define SW_TEMPLATE 0xac000000
if (KVM_GUEST_KSEGX((unsigned long)opc) == KVM_GUEST_KSEG0) {
paddr = kvm_mips_translate_guest_kseg0_to_hpa(vcpu,
(unsigned long)opc);
vaddr = kmap_atomic(pfn_to_page(PHYS_PFN(paddr)));
vaddr += paddr & ~PAGE_MASK;
memcpy(vaddr, (void *)&replace, sizeof(u32));
local_flush_icache_range((unsigned long)vaddr,
(unsigned long)vaddr + 32);
kunmap_atomic(vaddr);
} else if (KVM_GUEST_KSEGX((unsigned long) opc) == KVM_GUEST_KSEG23) {
local_irq_save(flags);
memcpy((void *)opc, (void *)&replace, sizeof(u32));
local_flush_icache_range((unsigned long)opc,
(unsigned long)opc + 32);
local_irq_restore(flags);
} else {
kvm_err("%s: Invalid address: %p\n", __func__, opc);
return -EFAULT;
}
int kvm_mips_trans_cache_index(uint32_t inst, uint32_t *opc,
return 0;
}
int kvm_mips_trans_cache_index(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu)
{
int result = 0;
unsigned long kseg0_opc;
uint32_t synci_inst = 0x0;
union mips_instruction nop_inst = { 0 };
/* Replace the CACHE instruction, with a NOP */
kseg0_opc =
CKSEG0ADDR(kvm_mips_translate_guest_kseg0_to_hpa
(vcpu, (unsigned long) opc));
memcpy((void *)kseg0_opc, (void *)&synci_inst, sizeof(uint32_t));
local_flush_icache_range(kseg0_opc, kseg0_opc + 32);
return result;
return kvm_mips_trans_replace(vcpu, opc, nop_inst);
}
/*
* Address based CACHE instructions are transformed into synci(s). A little
* heavy for just D-cache invalidates, but avoids an expensive trap
*/
int kvm_mips_trans_cache_va(uint32_t inst, uint32_t *opc,
int kvm_mips_trans_cache_va(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu)
{
int result = 0;
unsigned long kseg0_opc;
uint32_t synci_inst = SYNCI_TEMPLATE, base, offset;
union mips_instruction synci_inst = { 0 };
base = (inst >> 21) & 0x1f;
offset = inst & 0xffff;
synci_inst |= (base << 21);
synci_inst |= offset;
synci_inst.i_format.opcode = bcond_op;
synci_inst.i_format.rs = inst.i_format.rs;
synci_inst.i_format.rt = synci_op;
if (cpu_has_mips_r6)
synci_inst.i_format.simmediate = inst.spec3_format.simmediate;
else
synci_inst.i_format.simmediate = inst.i_format.simmediate;
kseg0_opc =
CKSEG0ADDR(kvm_mips_translate_guest_kseg0_to_hpa
(vcpu, (unsigned long) opc));
memcpy((void *)kseg0_opc, (void *)&synci_inst, sizeof(uint32_t));
local_flush_icache_range(kseg0_opc, kseg0_opc + 32);
return result;
return kvm_mips_trans_replace(vcpu, opc, synci_inst);
}
int kvm_mips_trans_mfc0(uint32_t inst, uint32_t *opc, struct kvm_vcpu *vcpu)
int kvm_mips_trans_mfc0(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu)
{
int32_t rt, rd, sel;
uint32_t mfc0_inst;
unsigned long kseg0_opc, flags;
union mips_instruction mfc0_inst = { 0 };
u32 rd, sel;
rt = (inst >> 16) & 0x1f;
rd = (inst >> 11) & 0x1f;
sel = inst & 0x7;
rd = inst.c0r_format.rd;
sel = inst.c0r_format.sel;
if ((rd == MIPS_CP0_ERRCTL) && (sel == 0)) {
mfc0_inst = CLEAR_TEMPLATE;
mfc0_inst |= ((rt & 0x1f) << 16);
if (rd == MIPS_CP0_ERRCTL && sel == 0) {
mfc0_inst.r_format.opcode = spec_op;
mfc0_inst.r_format.rd = inst.c0r_format.rt;
mfc0_inst.r_format.func = add_op;
} else {
mfc0_inst = LW_TEMPLATE;
mfc0_inst |= ((rt & 0x1f) << 16);
mfc0_inst |= offsetof(struct kvm_mips_commpage,
cop0.reg[rd][sel]);
mfc0_inst.i_format.opcode = lw_op;
mfc0_inst.i_format.rt = inst.c0r_format.rt;
mfc0_inst.i_format.simmediate = KVM_GUEST_COMMPAGE_ADDR |
offsetof(struct kvm_mips_commpage, cop0.reg[rd][sel]);
#ifdef CONFIG_CPU_BIG_ENDIAN
if (sizeof(vcpu->arch.cop0->reg[0][0]) == 8)
mfc0_inst.i_format.simmediate |= 4;
#endif
}
if (KVM_GUEST_KSEGX(opc) == KVM_GUEST_KSEG0) {
kseg0_opc =
CKSEG0ADDR(kvm_mips_translate_guest_kseg0_to_hpa
(vcpu, (unsigned long) opc));
memcpy((void *)kseg0_opc, (void *)&mfc0_inst, sizeof(uint32_t));
local_flush_icache_range(kseg0_opc, kseg0_opc + 32);
} else if (KVM_GUEST_KSEGX((unsigned long) opc) == KVM_GUEST_KSEG23) {
local_irq_save(flags);
memcpy((void *)opc, (void *)&mfc0_inst, sizeof(uint32_t));
local_flush_icache_range((unsigned long)opc,
(unsigned long)opc + 32);
local_irq_restore(flags);
} else {
kvm_err("%s: Invalid address: %p\n", __func__, opc);
return -EFAULT;
}
return 0;
return kvm_mips_trans_replace(vcpu, opc, mfc0_inst);
}
int kvm_mips_trans_mtc0(uint32_t inst, uint32_t *opc, struct kvm_vcpu *vcpu)
int kvm_mips_trans_mtc0(union mips_instruction inst, u32 *opc,
struct kvm_vcpu *vcpu)
{
int32_t rt, rd, sel;
uint32_t mtc0_inst = SW_TEMPLATE;
unsigned long kseg0_opc, flags;
union mips_instruction mtc0_inst = { 0 };
u32 rd, sel;
rt = (inst >> 16) & 0x1f;
rd = (inst >> 11) & 0x1f;
sel = inst & 0x7;
rd = inst.c0r_format.rd;
sel = inst.c0r_format.sel;
mtc0_inst |= ((rt & 0x1f) << 16);
mtc0_inst |= offsetof(struct kvm_mips_commpage, cop0.reg[rd][sel]);
mtc0_inst.i_format.opcode = sw_op;
mtc0_inst.i_format.rt = inst.c0r_format.rt;
mtc0_inst.i_format.simmediate = KVM_GUEST_COMMPAGE_ADDR |
offsetof(struct kvm_mips_commpage, cop0.reg[rd][sel]);
#ifdef CONFIG_CPU_BIG_ENDIAN
if (sizeof(vcpu->arch.cop0->reg[0][0]) == 8)
mtc0_inst.i_format.simmediate |= 4;
#endif
if (KVM_GUEST_KSEGX(opc) == KVM_GUEST_KSEG0) {
kseg0_opc =
CKSEG0ADDR(kvm_mips_translate_guest_kseg0_to_hpa
(vcpu, (unsigned long) opc));
memcpy((void *)kseg0_opc, (void *)&mtc0_inst, sizeof(uint32_t));
local_flush_icache_range(kseg0_opc, kseg0_opc + 32);
} else if (KVM_GUEST_KSEGX((unsigned long) opc) == KVM_GUEST_KSEG23) {
local_irq_save(flags);
memcpy((void *)opc, (void *)&mtc0_inst, sizeof(uint32_t));
local_flush_icache_range((unsigned long)opc,
(unsigned long)opc + 32);
local_irq_restore(flags);
} else {
kvm_err("%s: Invalid address: %p\n", __func__, opc);
return -EFAULT;
}
return 0;
return kvm_mips_trans_replace(vcpu, opc, mtc0_inst);
}

485
arch/mips/kvm/emulate.c
View File

File diff suppressed because it is too large Load Diff

701
arch/mips/kvm/entry.c Normal file
View File

@ -0,0 +1,701 @@
/*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file "COPYING" in the main directory of this archive
* for more details.
*
* Generation of main entry point for the guest, exception handling.
*
* Copyright (C) 2012 MIPS Technologies, Inc.
* Authors: Sanjay Lal <sanjayl@kymasys.com>
*
* Copyright (C) 2016 Imagination Technologies Ltd.
*/
#include <linux/kvm_host.h>
#include <asm/msa.h>
#include <asm/setup.h>
#include <asm/uasm.h>
/* Register names */
#define ZERO 0
#define AT 1
#define V0 2
#define V1 3
#define A0 4
#define A1 5
#if _MIPS_SIM == _MIPS_SIM_ABI32
#define T0 8
#define T1 9
#define T2 10
#define T3 11
#endif /* _MIPS_SIM == _MIPS_SIM_ABI32 */
#if _MIPS_SIM == _MIPS_SIM_ABI64 || _MIPS_SIM == _MIPS_SIM_NABI32
#define T0 12
#define T1 13
#define T2 14
#define T3 15
#endif /* _MIPS_SIM == _MIPS_SIM_ABI64 || _MIPS_SIM == _MIPS_SIM_NABI32 */
#define S0 16
#define S1 17
#define T9 25
#define K0 26
#define K1 27
#define GP 28
#define SP 29
#define RA 31
/* Some CP0 registers */
#define C0_HWRENA 7, 0
#define C0_BADVADDR 8, 0
#define C0_ENTRYHI 10, 0
#define C0_STATUS 12, 0
#define C0_CAUSE 13, 0
#define C0_EPC 14, 0
#define C0_EBASE 15, 1
#define C0_CONFIG5 16, 5
#define C0_DDATA_LO 28, 3
#define C0_ERROREPC 30, 0
#define CALLFRAME_SIZ 32
#ifdef CONFIG_64BIT
#define ST0_KX_IF_64 ST0_KX
#else
#define ST0_KX_IF_64 0
#endif
static unsigned int scratch_vcpu[2] = { C0_DDATA_LO };
static unsigned int scratch_tmp[2] = { C0_ERROREPC };
enum label_id {
label_fpu_1 = 1,
label_msa_1,
label_return_to_host,
label_kernel_asid,
label_exit_common,
};
UASM_L_LA(_fpu_1)
UASM_L_LA(_msa_1)
UASM_L_LA(_return_to_host)
UASM_L_LA(_kernel_asid)
UASM_L_LA(_exit_common)
static void *kvm_mips_build_enter_guest(void *addr);
static void *kvm_mips_build_ret_from_exit(void *addr);
static void *kvm_mips_build_ret_to_guest(void *addr);
static void *kvm_mips_build_ret_to_host(void *addr);
/**
* kvm_mips_entry_setup() - Perform global setup for entry code.
*
* Perform global setup for entry code, such as choosing a scratch register.
*
* Returns: 0 on success.
* -errno on failure.
*/
int kvm_mips_entry_setup(void)
{
/*
* We prefer to use KScratchN registers if they are available over the
* defaults above, which may not work on all cores.
*/
unsigned int kscratch_mask = cpu_data[0].kscratch_mask & 0xfc;
/* Pick a scratch register for storing VCPU */
if (kscratch_mask) {
scratch_vcpu[0] = 31;
scratch_vcpu[1] = ffs(kscratch_mask) - 1;
kscratch_mask &= ~BIT(scratch_vcpu[1]);
}
/* Pick a scratch register to use as a temp for saving state */
if (kscratch_mask) {
scratch_tmp[0] = 31;
scratch_tmp[1] = ffs(kscratch_mask) - 1;
kscratch_mask &= ~BIT(scratch_tmp[1]);
}
return 0;
}
static void kvm_mips_build_save_scratch(u32 **p, unsigned int tmp,
unsigned int frame)
{
/* Save the VCPU scratch register value in cp0_epc of the stack frame */
UASM_i_MFC0(p, tmp, scratch_vcpu[0], scratch_vcpu[1]);
UASM_i_SW(p, tmp, offsetof(struct pt_regs, cp0_epc), frame);
/* Save the temp scratch register value in cp0_cause of stack frame */
if (scratch_tmp[0] == 31) {
UASM_i_MFC0(p, tmp, scratch_tmp[0], scratch_tmp[1]);
UASM_i_SW(p, tmp, offsetof(struct pt_regs, cp0_cause), frame);
}
}
static void kvm_mips_build_restore_scratch(u32 **p, unsigned int tmp,
unsigned int frame)
{
/*
* Restore host scratch register values saved by
* kvm_mips_build_save_scratch().
*/
UASM_i_LW(p, tmp, offsetof(struct pt_regs, cp0_epc), frame);
UASM_i_MTC0(p, tmp, scratch_vcpu[0], scratch_vcpu[1]);
if (scratch_tmp[0] == 31) {
UASM_i_LW(p, tmp, offsetof(struct pt_regs, cp0_cause), frame);
UASM_i_MTC0(p, tmp, scratch_tmp[0], scratch_tmp[1]);
}
}
/**
* build_set_exc_base() - Assemble code to write exception base address.
* @p: Code buffer pointer.
* @reg: Source register (generated code may set WG bit in @reg).
*
* Assemble code to modify the exception base address in the EBase register,
* using the appropriately sized access and setting the WG bit if necessary.
*/
static inline void build_set_exc_base(u32 **p, unsigned int reg)
{
if (cpu_has_ebase_wg) {
/* Set WG so that all the bits get written */
uasm_i_ori(p, reg, reg, MIPS_EBASE_WG);
UASM_i_MTC0(p, reg, C0_EBASE);
} else {
uasm_i_mtc0(p, reg, C0_EBASE);
}
}
/**
* kvm_mips_build_vcpu_run() - Assemble function to start running a guest VCPU.
* @addr: Address to start writing code.
*
* Assemble the start of the vcpu_run function to run a guest VCPU. The function
* conforms to the following prototype:
*
* int vcpu_run(struct kvm_run *run, struct kvm_vcpu *vcpu);
*
* The exit from the guest and return to the caller is handled by the code
* generated by kvm_mips_build_ret_to_host().
*
* Returns: Next address after end of written function.
*/
void *kvm_mips_build_vcpu_run(void *addr)
{
u32 *p = addr;
unsigned int i;
/*
* A0: run
* A1: vcpu
*/
/* k0/k1 not being used in host kernel context */
UASM_i_ADDIU(&p, K1, SP, -(int)sizeof(struct pt_regs));
for (i = 16; i < 32; ++i) {
if (i == 24)
i = 28;
UASM_i_SW(&p, i, offsetof(struct pt_regs, regs[i]), K1);
}
/* Save host status */
uasm_i_mfc0(&p, V0, C0_STATUS);
UASM_i_SW(&p, V0, offsetof(struct pt_regs, cp0_status), K1);
/* Save scratch registers, will be used to store pointer to vcpu etc */
kvm_mips_build_save_scratch(&p, V1, K1);
/* VCPU scratch register has pointer to vcpu */
UASM_i_MTC0(&p, A1, scratch_vcpu[0], scratch_vcpu[1]);
/* Offset into vcpu->arch */
UASM_i_ADDIU(&p, K1, A1, offsetof(struct kvm_vcpu, arch));
/*
* Save the host stack to VCPU, used for exception processing
* when we exit from the Guest
*/
UASM_i_SW(&p, SP, offsetof(struct kvm_vcpu_arch, host_stack), K1);
/* Save the kernel gp as well */
UASM_i_SW(&p, GP, offsetof(struct kvm_vcpu_arch, host_gp), K1);
/*
* Setup status register for running the guest in UM, interrupts
* are disabled
*/
UASM_i_LA(&p, K0, ST0_EXL | KSU_USER | ST0_BEV | ST0_KX_IF_64);
uasm_i_mtc0(&p, K0, C0_STATUS);
uasm_i_ehb(&p);
/* load up the new EBASE */
UASM_i_LW(&p, K0, offsetof(struct kvm_vcpu_arch, guest_ebase), K1);
build_set_exc_base(&p, K0);
/*
* Now that the new EBASE has been loaded, unset BEV, set
* interrupt mask as it was but make sure that timer interrupts
* are enabled
*/
uasm_i_addiu(&p, K0, ZERO, ST0_EXL | KSU_USER | ST0_IE | ST0_KX_IF_64);
uasm_i_andi(&p, V0, V0, ST0_IM);
uasm_i_or(&p, K0, K0, V0);
uasm_i_mtc0(&p, K0, C0_STATUS);
uasm_i_ehb(&p);
p = kvm_mips_build_enter_guest(p);
return p;
}
/**
* kvm_mips_build_enter_guest() - Assemble code to resume guest execution.
* @addr: Address to start writing code.
*
* Assemble the code to resume guest execution. This code is common between the
* initial entry into the guest from the host, and returning from the exit
* handler back to the guest.
*
* Returns: Next address after end of written function.
*/
static void *kvm_mips_build_enter_guest(void *addr)
{
u32 *p = addr;
unsigned int i;
struct uasm_label labels[2];
struct uasm_reloc relocs[2];
struct uasm_label *l = labels;
struct uasm_reloc *r = relocs;
memset(labels, 0, sizeof(labels));
memset(relocs, 0, sizeof(relocs));
/* Set Guest EPC */
UASM_i_LW(&p, T0, offsetof(struct kvm_vcpu_arch, pc), K1);
UASM_i_MTC0(&p, T0, C0_EPC);
/* Set the ASID for the Guest Kernel */
UASM_i_LW(&p, T0, offsetof(struct kvm_vcpu_arch, cop0), K1);
UASM_i_LW(&p, T0, offsetof(struct mips_coproc, reg[MIPS_CP0_STATUS][0]),
T0);
uasm_i_andi(&p, T0, T0, KSU_USER | ST0_ERL | ST0_EXL);
uasm_i_xori(&p, T0, T0, KSU_USER);
uasm_il_bnez(&p, &r, T0, label_kernel_asid);
UASM_i_ADDIU(&p, T1, K1,
offsetof(struct kvm_vcpu_arch, guest_kernel_asid));
/* else user */
UASM_i_ADDIU(&p, T1, K1,
offsetof(struct kvm_vcpu_arch, guest_user_asid));
uasm_l_kernel_asid(&l, p);
/* t1: contains the base of the ASID array, need to get the cpu id */
/* smp_processor_id */
uasm_i_lw(&p, T2, offsetof(struct thread_info, cpu), GP);
/* x4 */
uasm_i_sll(&p, T2, T2, 2);
UASM_i_ADDU(&p, T3, T1, T2);
uasm_i_lw(&p, K0, 0, T3);
#ifdef CONFIG_MIPS_ASID_BITS_VARIABLE
/* x sizeof(struct cpuinfo_mips)/4 */
uasm_i_addiu(&p, T3, ZERO, sizeof(struct cpuinfo_mips)/4);
uasm_i_mul(&p, T2, T2, T3);
UASM_i_LA_mostly(&p, AT, (long)&cpu_data[0].asid_mask);
UASM_i_ADDU(&p, AT, AT, T2);
UASM_i_LW(&p, T2, uasm_rel_lo((long)&cpu_data[0].asid_mask), AT);
uasm_i_and(&p, K0, K0, T2);
#else
uasm_i_andi(&p, K0, K0, MIPS_ENTRYHI_ASID);
#endif
uasm_i_mtc0(&p, K0, C0_ENTRYHI);
uasm_i_ehb(&p);
/* Disable RDHWR access */
uasm_i_mtc0(&p, ZERO, C0_HWRENA);
/* load the guest context from VCPU and return */
for (i = 1; i < 32; ++i) {
/* Guest k0/k1 loaded later */
if (i == K0 || i == K1)
continue;
UASM_i_LW(&p, i, offsetof(struct kvm_vcpu_arch, gprs[i]), K1);
}
#ifndef CONFIG_CPU_MIPSR6
/* Restore hi/lo */
UASM_i_LW(&p, K0, offsetof(struct kvm_vcpu_arch, hi), K1);
uasm_i_mthi(&p, K0);
UASM_i_LW(&p, K0, offsetof(struct kvm_vcpu_arch, lo), K1);
uasm_i_mtlo(&p, K0);
#endif
/* Restore the guest's k0/k1 registers */
UASM_i_LW(&p, K0, offsetof(struct kvm_vcpu_arch, gprs[K0]), K1);
UASM_i_LW(&p, K1, offsetof(struct kvm_vcpu_arch, gprs[K1]), K1);
/* Jump to guest */
uasm_i_eret(&p);
uasm_resolve_relocs(relocs, labels);
return p;
}
/**
* kvm_mips_build_exception() - Assemble first level guest exception handler.
* @addr: Address to start writing code.
* @handler: Address of common handler (within range of @addr).
*
* Assemble exception vector code for guest execution. The generated vector will
* branch to the common exception handler generated by kvm_mips_build_exit().
*
* Returns: Next address after end of written function.
*/
void *kvm_mips_build_exception(void *addr, void *handler)
{
u32 *p = addr;
struct uasm_label labels[2];
struct uasm_reloc relocs[2];
struct uasm_label *l = labels;
struct uasm_reloc *r = relocs;
memset(labels, 0, sizeof(labels));
memset(relocs, 0, sizeof(relocs));
/* Save guest k1 into scratch register */
UASM_i_MTC0(&p, K1, scratch_tmp[0], scratch_tmp[1]);
/* Get the VCPU pointer from the VCPU scratch register */
UASM_i_MFC0(&p, K1, scratch_vcpu[0], scratch_vcpu[1]);
UASM_i_ADDIU(&p, K1, K1, offsetof(struct kvm_vcpu, arch));
/* Save guest k0 into VCPU structure */
UASM_i_SW(&p, K0, offsetof(struct kvm_vcpu_arch, gprs[K0]), K1);
/* Branch to the common handler */
uasm_il_b(&p, &r, label_exit_common);
uasm_i_nop(&p);
uasm_l_exit_common(&l, handler);
uasm_resolve_relocs(relocs, labels);
return p;
}
/**
* kvm_mips_build_exit() - Assemble common guest exit handler.
* @addr: Address to start writing code.
*
* Assemble the generic guest exit handling code. This is called by the
* exception vectors (generated by kvm_mips_build_exception()), and calls
* kvm_mips_handle_exit(), then either resumes the guest or returns to the host
* depending on the return value.
*
* Returns: Next address after end of written function.
*/
void *kvm_mips_build_exit(void *addr)
{
u32 *p = addr;
unsigned int i;
struct uasm_label labels[3];
struct uasm_reloc relocs[3];
struct uasm_label *l = labels;
struct uasm_reloc *r = relocs;
memset(labels, 0, sizeof(labels));
memset(relocs, 0, sizeof(relocs));
/*
* Generic Guest exception handler. We end up here when the guest
* does something that causes a trap to kernel mode.
*
* Both k0/k1 registers will have already been saved (k0 into the vcpu
* structure, and k1 into the scratch_tmp register).
*
* The k1 register will already contain the kvm_vcpu_arch pointer.
*/
/* Start saving Guest context to VCPU */
for (i = 0; i < 32; ++i) {
/* Guest k0/k1 saved later */
if (i == K0 || i == K1)
continue;
UASM_i_SW(&p, i, offsetof(struct kvm_vcpu_arch, gprs[i]), K1);
}
#ifndef CONFIG_CPU_MIPSR6
/* We need to save hi/lo and restore them on the way out */
uasm_i_mfhi(&p, T0);
UASM_i_SW(&p, T0, offsetof(struct kvm_vcpu_arch, hi), K1);
uasm_i_mflo(&p, T0);
UASM_i_SW(&p, T0, offsetof(struct kvm_vcpu_arch, lo), K1);
#endif
/* Finally save guest k1 to VCPU */
uasm_i_ehb(&p);
UASM_i_MFC0(&p, T0, scratch_tmp[0], scratch_tmp[1]);
UASM_i_SW(&p, T0, offsetof(struct kvm_vcpu_arch, gprs[K1]), K1);
/* Now that context has been saved, we can use other registers */
/* Restore vcpu */
UASM_i_MFC0(&p, A1, scratch_vcpu[0], scratch_vcpu[1]);
uasm_i_move(&p, S1, A1);
/* Restore run (vcpu->run) */
UASM_i_LW(&p, A0, offsetof(struct kvm_vcpu, run), A1);
/* Save pointer to run in s0, will be saved by the compiler */
uasm_i_move(&p, S0, A0);
/*
* Save Host level EPC, BadVaddr and Cause to VCPU, useful to process
* the exception
*/
UASM_i_MFC0(&p, K0, C0_EPC);
UASM_i_SW(&p, K0, offsetof(struct kvm_vcpu_arch, pc), K1);
UASM_i_MFC0(&p, K0, C0_BADVADDR);
UASM_i_SW(&p, K0, offsetof(struct kvm_vcpu_arch, host_cp0_badvaddr),
K1);
uasm_i_mfc0(&p, K0, C0_CAUSE);
uasm_i_sw(&p, K0, offsetof(struct kvm_vcpu_arch, host_cp0_cause), K1);
/* Now restore the host state just enough to run the handlers */
/* Switch EBASE to the one used by Linux */
/* load up the host EBASE */
uasm_i_mfc0(&p, V0, C0_STATUS);
uasm_i_lui(&p, AT, ST0_BEV >> 16);
uasm_i_or(&p, K0, V0, AT);
uasm_i_mtc0(&p, K0, C0_STATUS);
uasm_i_ehb(&p);
UASM_i_LA_mostly(&p, K0, (long)&ebase);
UASM_i_LW(&p, K0, uasm_rel_lo((long)&ebase), K0);
build_set_exc_base(&p, K0);
if (raw_cpu_has_fpu) {
/*
* If FPU is enabled, save FCR31 and clear it so that later
* ctc1's don't trigger FPE for pending exceptions.
*/
uasm_i_lui(&p, AT, ST0_CU1 >> 16);
uasm_i_and(&p, V1, V0, AT);
uasm_il_beqz(&p, &r, V1, label_fpu_1);
uasm_i_nop(&p);
uasm_i_cfc1(&p, T0, 31);
uasm_i_sw(&p, T0, offsetof(struct kvm_vcpu_arch, fpu.fcr31),
K1);
uasm_i_ctc1(&p, ZERO, 31);
uasm_l_fpu_1(&l, p);
}
if (cpu_has_msa) {
/*
* If MSA is enabled, save MSACSR and clear it so that later
* instructions don't trigger MSAFPE for pending exceptions.
*/
uasm_i_mfc0(&p, T0, C0_CONFIG5);
uasm_i_ext(&p, T0, T0, 27, 1); /* MIPS_CONF5_MSAEN */
uasm_il_beqz(&p, &r, T0, label_msa_1);
uasm_i_nop(&p);
uasm_i_cfcmsa(&p, T0, MSA_CSR);
uasm_i_sw(&p, T0, offsetof(struct kvm_vcpu_arch, fpu.msacsr),
K1);
uasm_i_ctcmsa(&p, MSA_CSR, ZERO);
uasm_l_msa_1(&l, p);
}
/* Now that the new EBASE has been loaded, unset BEV and KSU_USER */
uasm_i_addiu(&p, AT, ZERO, ~(ST0_EXL | KSU_USER | ST0_IE));
uasm_i_and(&p, V0, V0, AT);
uasm_i_lui(&p, AT, ST0_CU0 >> 16);
uasm_i_or(&p, V0, V0, AT);
uasm_i_mtc0(&p, V0, C0_STATUS);
uasm_i_ehb(&p);
/* Load up host GP */
UASM_i_LW(&p, GP, offsetof(struct kvm_vcpu_arch, host_gp), K1);
/* Need a stack before we can jump to "C" */
UASM_i_LW(&p, SP, offsetof(struct kvm_vcpu_arch, host_stack), K1);
/* Saved host state */
UASM_i_ADDIU(&p, SP, SP, -(int)sizeof(struct pt_regs));
/*
* XXXKYMA do we need to load the host ASID, maybe not because the
* kernel entries are marked GLOBAL, need to verify
*/
/* Restore host scratch registers, as we'll have clobbered them */
kvm_mips_build_restore_scratch(&p, K0, SP);
/* Restore RDHWR access */
UASM_i_LA_mostly(&p, K0, (long)&hwrena);
uasm_i_lw(&p, K0, uasm_rel_lo((long)&hwrena), K0);
uasm_i_mtc0(&p, K0, C0_HWRENA);
/* Jump to handler */
/*
* XXXKYMA: not sure if this is safe, how large is the stack??
* Now jump to the kvm_mips_handle_exit() to see if we can deal
* with this in the kernel
*/
UASM_i_LA(&p, T9, (unsigned long)kvm_mips_handle_exit);
uasm_i_jalr(&p, RA, T9);
UASM_i_ADDIU(&p, SP, SP, -CALLFRAME_SIZ);
uasm_resolve_relocs(relocs, labels);
p = kvm_mips_build_ret_from_exit(p);
return p;
}
/**
* kvm_mips_build_ret_from_exit() - Assemble guest exit return handler.
* @addr: Address to start writing code.
*
* Assemble the code to handle the return from kvm_mips_handle_exit(), either
* resuming the guest or returning to the host depending on the return value.
*
* Returns: Next address after end of written function.
*/
static void *kvm_mips_build_ret_from_exit(void *addr)
{
u32 *p = addr;
struct uasm_label labels[2];
struct uasm_reloc relocs[2];
struct uasm_label *l = labels;
struct uasm_reloc *r = relocs;
memset(labels, 0, sizeof(labels));
memset(relocs, 0, sizeof(relocs));
/* Return from handler Make sure interrupts are disabled */
uasm_i_di(&p, ZERO);
uasm_i_ehb(&p);
/*
* XXXKYMA: k0/k1 could have been blown away if we processed
* an exception while we were handling the exception from the
* guest, reload k1
*/
uasm_i_move(&p, K1, S1);
UASM_i_ADDIU(&p, K1, K1, offsetof(struct kvm_vcpu, arch));
/*
* Check return value, should tell us if we are returning to the
* host (handle I/O etc)or resuming the guest
*/
uasm_i_andi(&p, T0, V0, RESUME_HOST);
uasm_il_bnez(&p, &r, T0, label_return_to_host);
uasm_i_nop(&p);
p = kvm_mips_build_ret_to_guest(p);
uasm_l_return_to_host(&l, p);
p = kvm_mips_build_ret_to_host(p);
uasm_resolve_relocs(relocs, labels);
return p;
}
/**
* kvm_mips_build_ret_to_guest() - Assemble code to return to the guest.
* @addr: Address to start writing code.
*
* Assemble the code to handle return from the guest exit handler
* (kvm_mips_handle_exit()) back to the guest.
*
* Returns: Next address after end of written function.
*/
static void *kvm_mips_build_ret_to_guest(void *addr)
{
u32 *p = addr;
/* Put the saved pointer to vcpu (s1) back into the scratch register */
UASM_i_MTC0(&p, S1, scratch_vcpu[0], scratch_vcpu[1]);
/* Load up the Guest EBASE to minimize the window where BEV is set */
UASM_i_LW(&p, T0, offsetof(struct kvm_vcpu_arch, guest_ebase), K1);
/* Switch EBASE back to the one used by KVM */
uasm_i_mfc0(&p, V1, C0_STATUS);
uasm_i_lui(&p, AT, ST0_BEV >> 16);
uasm_i_or(&p, K0, V1, AT);
uasm_i_mtc0(&p, K0, C0_STATUS);
uasm_i_ehb(&p);
build_set_exc_base(&p, T0);
/* Setup status register for running guest in UM */
uasm_i_ori(&p, V1, V1, ST0_EXL | KSU_USER | ST0_IE);
UASM_i_LA(&p, AT, ~(ST0_CU0 | ST0_MX));
uasm_i_and(&p, V1, V1, AT);
uasm_i_mtc0(&p, V1, C0_STATUS);
uasm_i_ehb(&p);
p = kvm_mips_build_enter_guest(p);
return p;
}
/**
* kvm_mips_build_ret_to_host() - Assemble code to return to the host.
* @addr: Address to start writing code.
*
* Assemble the code to handle return from the guest exit handler
* (kvm_mips_handle_exit()) back to the host, i.e. to the caller of the vcpu_run
* function generated by kvm_mips_build_vcpu_run().
*
* Returns: Next address after end of written function.
*/
static void *kvm_mips_build_ret_to_host(void *addr)
{
u32 *p = addr;
unsigned int i;
/* EBASE is already pointing to Linux */
UASM_i_LW(&p, K1, offsetof(struct kvm_vcpu_arch, host_stack), K1);
UASM_i_ADDIU(&p, K1, K1, -(int)sizeof(struct pt_regs));
/*
* r2/v0 is the return code, shift it down by 2 (arithmetic)
* to recover the err code
*/
uasm_i_sra(&p, K0, V0, 2);
uasm_i_move(&p, V0, K0);
/* Load context saved on the host stack */
for (i = 16; i < 31; ++i) {
if (i == 24)
i = 28;
UASM_i_LW(&p, i, offsetof(struct pt_regs, regs[i]), K1);
}
/* Restore RDHWR access */
UASM_i_LA_mostly(&p, K0, (long)&hwrena);
uasm_i_lw(&p, K0, uasm_rel_lo((long)&hwrena), K0);
uasm_i_mtc0(&p, K0, C0_HWRENA);
/* Restore RA, which is the address we will return to */
UASM_i_LW(&p, RA, offsetof(struct pt_regs, regs[RA]), K1);
uasm_i_jr(&p, RA);
uasm_i_nop(&p);
return p;
}

7
arch/mips/kvm/fpu.S
View File

@ -14,13 +14,16 @@
#include <asm/mipsregs.h>
#include <asm/regdef.h>
/* preprocessor replaces the fp in ".set fp=64" with $30 otherwise */
#undef fp
.set noreorder
.set noat
LEAF(__kvm_save_fpu)
.set push
.set mips64r2
SET_HARDFLOAT
.set fp=64
mfc0 t0, CP0_STATUS
sll t0, t0, 5 # is Status.FR set?
bgez t0, 1f # no: skip odd doubles
@ -63,8 +66,8 @@ LEAF(__kvm_save_fpu)
LEAF(__kvm_restore_fpu)
.set push
.set mips64r2
SET_HARDFLOAT
.set fp=64
mfc0 t0, CP0_STATUS
sll t0, t0, 5 # is Status.FR set?
bgez t0, 1f # no: skip odd doubles

12
arch/mips/kvm/interrupt.c
View File

@ -22,12 +22,12 @@
#include "interrupt.h"
void kvm_mips_queue_irq(struct kvm_vcpu *vcpu, uint32_t priority)
void kvm_mips_queue_irq(struct kvm_vcpu *vcpu, unsigned int priority)
{
set_bit(priority, &vcpu->arch.pending_exceptions);
}
void kvm_mips_dequeue_irq(struct kvm_vcpu *vcpu, uint32_t priority)
void kvm_mips_dequeue_irq(struct kvm_vcpu *vcpu, unsigned int priority)
{
clear_bit(priority, &vcpu->arch.pending_exceptions);
}
@ -114,10 +114,10 @@ void kvm_mips_dequeue_io_int_cb(struct kvm_vcpu *vcpu,
/* Deliver the interrupt of the corresponding priority, if possible. */
int kvm_mips_irq_deliver_cb(struct kvm_vcpu *vcpu, unsigned int priority,
uint32_t cause)
u32 cause)
{
int allowed = 0;
uint32_t exccode;
u32 exccode;
struct kvm_vcpu_arch *arch = &vcpu->arch;
struct mips_coproc *cop0 = vcpu->arch.cop0;
@ -196,12 +196,12 @@ int kvm_mips_irq_deliver_cb(struct kvm_vcpu *vcpu, unsigned int priority,
}
int kvm_mips_irq_clear_cb(struct kvm_vcpu *vcpu, unsigned int priority,
uint32_t cause)
u32 cause)
{
return 1;
}
void kvm_mips_deliver_interrupts(struct kvm_vcpu *vcpu, uint32_t cause)
void kvm_mips_deliver_interrupts(struct kvm_vcpu *vcpu, u32 cause)
{
unsigned long *pending = &vcpu->arch.pending_exceptions;
unsigned long *pending_clr = &vcpu->arch.pending_exceptions_clr;

14
arch/mips/kvm/interrupt.h
View File

@ -28,17 +28,13 @@
#define MIPS_EXC_MAX 12
/* XXXSL More to follow */
extern char __kvm_mips_vcpu_run_end[];
extern char mips32_exception[], mips32_exceptionEnd[];
extern char mips32_GuestException[], mips32_GuestExceptionEnd[];
#define C_TI (_ULCAST_(1) << 30)
#define KVM_MIPS_IRQ_DELIVER_ALL_AT_ONCE (0)
#define KVM_MIPS_IRQ_CLEAR_ALL_AT_ONCE (0)
void kvm_mips_queue_irq(struct kvm_vcpu *vcpu, uint32_t priority);
void kvm_mips_dequeue_irq(struct kvm_vcpu *vcpu, uint32_t priority);
void kvm_mips_queue_irq(struct kvm_vcpu *vcpu, unsigned int priority);
void kvm_mips_dequeue_irq(struct kvm_vcpu *vcpu, unsigned int priority);
int kvm_mips_pending_timer(struct kvm_vcpu *vcpu);
void kvm_mips_queue_timer_int_cb(struct kvm_vcpu *vcpu);
@ -48,7 +44,7 @@ void kvm_mips_queue_io_int_cb(struct kvm_vcpu *vcpu,
void kvm_mips_dequeue_io_int_cb(struct kvm_vcpu *vcpu,
struct kvm_mips_interrupt *irq);
int kvm_mips_irq_deliver_cb(struct kvm_vcpu *vcpu, unsigned int priority,
uint32_t cause);
u32 cause);
int kvm_mips_irq_clear_cb(struct kvm_vcpu *vcpu, unsigned int priority,
uint32_t cause);
void kvm_mips_deliver_interrupts(struct kvm_vcpu *vcpu, uint32_t cause);
u32 cause);
void kvm_mips_deliver_interrupts(struct kvm_vcpu *vcpu, u32 cause);

605
arch/mips/kvm/locore.S
View File

@ -1,605 +0,0 @@
/*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file "COPYING" in the main directory of this archive
* for more details.
*
* Main entry point for the guest, exception handling.
*
* Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
* Authors: Sanjay Lal <sanjayl@kymasys.com>
*/
#include <asm/asm.h>
#include <asm/asmmacro.h>
#include <asm/regdef.h>
#include <asm/mipsregs.h>
#include <asm/stackframe.h>
#include <asm/asm-offsets.h>
#define _C_LABEL(x) x
#define MIPSX(name) mips32_ ## name
#define CALLFRAME_SIZ 32
/*
* VECTOR
* exception vector entrypoint
*/
#define VECTOR(x, regmask) \
.ent _C_LABEL(x),0; \
EXPORT(x);
#define VECTOR_END(x) \
EXPORT(x);
/* Overload, Danger Will Robinson!! */
#define PT_HOST_USERLOCAL PT_EPC
#define CP0_DDATA_LO $28,3
/* Resume Flags */
#define RESUME_FLAG_HOST (1<<1) /* Resume host? */
#define RESUME_GUEST 0
#define RESUME_HOST RESUME_FLAG_HOST
/*
* __kvm_mips_vcpu_run: entry point to the guest
* a0: run
* a1: vcpu
*/
.set noreorder
FEXPORT(__kvm_mips_vcpu_run)
/* k0/k1 not being used in host kernel context */
INT_ADDIU k1, sp, -PT_SIZE
LONG_S $16, PT_R16(k1)
LONG_S $17, PT_R17(k1)
LONG_S $18, PT_R18(k1)
LONG_S $19, PT_R19(k1)
LONG_S $20, PT_R20(k1)
LONG_S $21, PT_R21(k1)
LONG_S $22, PT_R22(k1)
LONG_S $23, PT_R23(k1)
LONG_S $28, PT_R28(k1)
LONG_S $29, PT_R29(k1)
LONG_S $30, PT_R30(k1)
LONG_S $31, PT_R31(k1)
/* Save hi/lo */
mflo v0
LONG_S v0, PT_LO(k1)
mfhi v1
LONG_S v1, PT_HI(k1)
/* Save host status */
mfc0 v0, CP0_STATUS
LONG_S v0, PT_STATUS(k1)
/* Save DDATA_LO, will be used to store pointer to vcpu */
mfc0 v1, CP0_DDATA_LO
LONG_S v1, PT_HOST_USERLOCAL(k1)
/* DDATA_LO has pointer to vcpu */
mtc0 a1, CP0_DDATA_LO
/* Offset into vcpu->arch */
INT_ADDIU k1, a1, VCPU_HOST_ARCH
/*
* Save the host stack to VCPU, used for exception processing
* when we exit from the Guest
*/
LONG_S sp, VCPU_HOST_STACK(k1)
/* Save the kernel gp as well */
LONG_S gp, VCPU_HOST_GP(k1)
/*
* Setup status register for running the guest in UM, interrupts
* are disabled
*/
li k0, (ST0_EXL | KSU_USER | ST0_BEV)
mtc0 k0, CP0_STATUS
ehb
/* load up the new EBASE */
LONG_L k0, VCPU_GUEST_EBASE(k1)
mtc0 k0, CP0_EBASE
/*
* Now that the new EBASE has been loaded, unset BEV, set
* interrupt mask as it was but make sure that timer interrupts
* are enabled
*/
li k0, (ST0_EXL | KSU_USER | ST0_IE)
andi v0, v0, ST0_IM
or k0, k0, v0
mtc0 k0, CP0_STATUS
ehb
/* Set Guest EPC */
LONG_L t0, VCPU_PC(k1)
mtc0 t0, CP0_EPC
FEXPORT(__kvm_mips_load_asid)
/* Set the ASID for the Guest Kernel */
PTR_L t0, VCPU_COP0(k1)
LONG_L t0, COP0_STATUS(t0)
andi t0, KSU_USER | ST0_ERL | ST0_EXL
xori t0, KSU_USER
bnez t0, 1f /* If kernel */
INT_ADDIU t1, k1, VCPU_GUEST_KERNEL_ASID /* (BD) */
INT_ADDIU t1, k1, VCPU_GUEST_USER_ASID /* else user */
1:
/* t1: contains the base of the ASID array, need to get the cpu id */
LONG_L t2, TI_CPU($28) /* smp_processor_id */
INT_SLL t2, t2, 2 /* x4 */
REG_ADDU t3, t1, t2
LONG_L k0, (t3)
#ifdef CONFIG_MIPS_ASID_BITS_VARIABLE
li t3, CPUINFO_SIZE/4
mul t2, t2, t3 /* x sizeof(struct cpuinfo_mips)/4 */
LONG_L t2, (cpu_data + CPUINFO_ASID_MASK)(t2)
and k0, k0, t2
#else
andi k0, k0, MIPS_ENTRYHI_ASID
#endif
mtc0 k0, CP0_ENTRYHI
ehb
/* Disable RDHWR access */
mtc0 zero, CP0_HWRENA
.set noat
/* Now load up the Guest Context from VCPU */
LONG_L $1, VCPU_R1(k1)
LONG_L $2, VCPU_R2(k1)
LONG_L $3, VCPU_R3(k1)
LONG_L $4, VCPU_R4(k1)
LONG_L $5, VCPU_R5(k1)
LONG_L $6, VCPU_R6(k1)
LONG_L $7, VCPU_R7(k1)
LONG_L $8, VCPU_R8(k1)
LONG_L $9, VCPU_R9(k1)
LONG_L $10, VCPU_R10(k1)
LONG_L $11, VCPU_R11(k1)
LONG_L $12, VCPU_R12(k1)
LONG_L $13, VCPU_R13(k1)
LONG_L $14, VCPU_R14(k1)
LONG_L $15, VCPU_R15(k1)
LONG_L $16, VCPU_R16(k1)
LONG_L $17, VCPU_R17(k1)
LONG_L $18, VCPU_R18(k1)
LONG_L $19, VCPU_R19(k1)
LONG_L $20, VCPU_R20(k1)
LONG_L $21, VCPU_R21(k1)
LONG_L $22, VCPU_R22(k1)
LONG_L $23, VCPU_R23(k1)
LONG_L $24, VCPU_R24(k1)
LONG_L $25, VCPU_R25(k1)
/* k0/k1 loaded up later */
LONG_L $28, VCPU_R28(k1)
LONG_L $29, VCPU_R29(k1)
LONG_L $30, VCPU_R30(k1)
LONG_L $31, VCPU_R31(k1)
/* Restore hi/lo */
LONG_L k0, VCPU_LO(k1)
mtlo k0
LONG_L k0, VCPU_HI(k1)
mthi k0
FEXPORT(__kvm_mips_load_k0k1)
/* Restore the guest's k0/k1 registers */
LONG_L k0, VCPU_R26(k1)
LONG_L k1, VCPU_R27(k1)
/* Jump to guest */
eret
EXPORT(__kvm_mips_vcpu_run_end)
VECTOR(MIPSX(exception), unknown)
/* Find out what mode we came from and jump to the proper handler. */
mtc0 k0, CP0_ERROREPC #01: Save guest k0
ehb #02:
mfc0 k0, CP0_EBASE #02: Get EBASE
INT_SRL k0, k0, 10 #03: Get rid of CPUNum
INT_SLL k0, k0, 10 #04
LONG_S k1, 0x3000(k0) #05: Save k1 @ offset 0x3000
INT_ADDIU k0, k0, 0x2000 #06: Exception handler is
# installed @ offset 0x2000
j k0 #07: jump to the function
nop #08: branch delay slot
VECTOR_END(MIPSX(exceptionEnd))
.end MIPSX(exception)
/*
* Generic Guest exception handler. We end up here when the guest
* does something that causes a trap to kernel mode.
*/
NESTED (MIPSX(GuestException), CALLFRAME_SIZ, ra)
/* Get the VCPU pointer from DDTATA_LO */
mfc0 k1, CP0_DDATA_LO
INT_ADDIU k1, k1, VCPU_HOST_ARCH
/* Start saving Guest context to VCPU */
LONG_S $0, VCPU_R0(k1)
LONG_S $1, VCPU_R1(k1)
LONG_S $2, VCPU_R2(k1)
LONG_S $3, VCPU_R3(k1)
LONG_S $4, VCPU_R4(k1)
LONG_S $5, VCPU_R5(k1)
LONG_S $6, VCPU_R6(k1)
LONG_S $7, VCPU_R7(k1)
LONG_S $8, VCPU_R8(k1)
LONG_S $9, VCPU_R9(k1)
LONG_S $10, VCPU_R10(k1)
LONG_S $11, VCPU_R11(k1)
LONG_S $12, VCPU_R12(k1)
LONG_S $13, VCPU_R13(k1)
LONG_S $14, VCPU_R14(k1)
LONG_S $15, VCPU_R15(k1)
LONG_S $16, VCPU_R16(k1)
LONG_S $17, VCPU_R17(k1)
LONG_S $18, VCPU_R18(k1)
LONG_S $19, VCPU_R19(k1)
LONG_S $20, VCPU_R20(k1)
LONG_S $21, VCPU_R21(k1)
LONG_S $22, VCPU_R22(k1)
LONG_S $23, VCPU_R23(k1)
LONG_S $24, VCPU_R24(k1)
LONG_S $25, VCPU_R25(k1)
/* Guest k0/k1 saved later */
LONG_S $28, VCPU_R28(k1)
LONG_S $29, VCPU_R29(k1)
LONG_S $30, VCPU_R30(k1)
LONG_S $31, VCPU_R31(k1)
.set at
/* We need to save hi/lo and restore them on the way out */
mfhi t0
LONG_S t0, VCPU_HI(k1)
mflo t0
LONG_S t0, VCPU_LO(k1)
/* Finally save guest k0/k1 to VCPU */
mfc0 t0, CP0_ERROREPC
LONG_S t0, VCPU_R26(k1)
/* Get GUEST k1 and save it in VCPU */
PTR_LI t1, ~0x2ff
mfc0 t0, CP0_EBASE
and t0, t0, t1
LONG_L t0, 0x3000(t0)
LONG_S t0, VCPU_R27(k1)
/* Now that context has been saved, we can use other registers */
/* Restore vcpu */
mfc0 a1, CP0_DDATA_LO
move s1, a1
/* Restore run (vcpu->run) */
LONG_L a0, VCPU_RUN(a1)
/* Save pointer to run in s0, will be saved by the compiler */
move s0, a0
/*
* Save Host level EPC, BadVaddr and Cause to VCPU, useful to
* process the exception
*/
mfc0 k0,CP0_EPC
LONG_S k0, VCPU_PC(k1)
mfc0 k0, CP0_BADVADDR
LONG_S k0, VCPU_HOST_CP0_BADVADDR(k1)
mfc0 k0, CP0_CAUSE
LONG_S k0, VCPU_HOST_CP0_CAUSE(k1)
mfc0 k0, CP0_ENTRYHI
LONG_S k0, VCPU_HOST_ENTRYHI(k1)
/* Now restore the host state just enough to run the handlers */
/* Switch EBASE to the one used by Linux */
/* load up the host EBASE */
mfc0 v0, CP0_STATUS
or k0, v0, ST0_BEV
mtc0 k0, CP0_STATUS
ehb
LONG_L k0, VCPU_HOST_EBASE(k1)
mtc0 k0,CP0_EBASE
/*
* If FPU is enabled, save FCR31 and clear it so that later ctc1's don't
* trigger FPE for pending exceptions.
*/
and v1, v0, ST0_CU1
beqz v1, 1f
nop
.set push
SET_HARDFLOAT
cfc1 t0, fcr31
sw t0, VCPU_FCR31(k1)
ctc1 zero,fcr31
.set pop
1:
#ifdef CONFIG_CPU_HAS_MSA
/*
* If MSA is enabled, save MSACSR and clear it so that later
* instructions don't trigger MSAFPE for pending exceptions.
*/
mfc0 t0, CP0_CONFIG3
ext t0, t0, 28, 1 /* MIPS_CONF3_MSAP */
beqz t0, 1f
nop
mfc0 t0, CP0_CONFIG5
ext t0, t0, 27, 1 /* MIPS_CONF5_MSAEN */
beqz t0, 1f
nop
_cfcmsa t0, MSA_CSR
sw t0, VCPU_MSA_CSR(k1)
_ctcmsa MSA_CSR, zero
1:
#endif
/* Now that the new EBASE has been loaded, unset BEV and KSU_USER */
and v0, v0, ~(ST0_EXL | KSU_USER | ST0_IE)
or v0, v0, ST0_CU0
mtc0 v0, CP0_STATUS
ehb
/* Load up host GP */
LONG_L gp, VCPU_HOST_GP(k1)
/* Need a stack before we can jump to "C" */
LONG_L sp, VCPU_HOST_STACK(k1)
/* Saved host state */
INT_ADDIU sp, sp, -PT_SIZE
/*
* XXXKYMA do we need to load the host ASID, maybe not because the
* kernel entries are marked GLOBAL, need to verify
*/
/* Restore host DDATA_LO */
LONG_L k0, PT_HOST_USERLOCAL(sp)
mtc0 k0, CP0_DDATA_LO
/* Restore RDHWR access */
PTR_LI k0, 0x2000000F
mtc0 k0, CP0_HWRENA
/* Jump to handler */
FEXPORT(__kvm_mips_jump_to_handler)
/*
* XXXKYMA: not sure if this is safe, how large is the stack??
* Now jump to the kvm_mips_handle_exit() to see if we can deal
* with this in the kernel
*/
PTR_LA t9, kvm_mips_handle_exit
jalr.hb t9
INT_ADDIU sp, sp, -CALLFRAME_SIZ /* BD Slot */
/* Return from handler Make sure interrupts are disabled */
di
ehb
/*
* XXXKYMA: k0/k1 could have been blown away if we processed
* an exception while we were handling the exception from the
* guest, reload k1
*/
move k1, s1
INT_ADDIU k1, k1, VCPU_HOST_ARCH
/*
* Check return value, should tell us if we are returning to the
* host (handle I/O etc)or resuming the guest
*/
andi t0, v0, RESUME_HOST
bnez t0, __kvm_mips_return_to_host
nop
__kvm_mips_return_to_guest:
/* Put the saved pointer to vcpu (s1) back into the DDATA_LO Register */
mtc0 s1, CP0_DDATA_LO
/* Load up the Guest EBASE to minimize the window where BEV is set */
LONG_L t0, VCPU_GUEST_EBASE(k1)
/* Switch EBASE back to the one used by KVM */
mfc0 v1, CP0_STATUS
or k0, v1, ST0_BEV
mtc0 k0, CP0_STATUS
ehb
mtc0 t0, CP0_EBASE
/* Setup status register for running guest in UM */
or v1, v1, (ST0_EXL | KSU_USER | ST0_IE)
and v1, v1, ~(ST0_CU0 | ST0_MX)
mtc0 v1, CP0_STATUS
ehb
/* Set Guest EPC */
LONG_L t0, VCPU_PC(k1)
mtc0 t0, CP0_EPC
/* Set the ASID for the Guest Kernel */
PTR_L t0, VCPU_COP0(k1)
LONG_L t0, COP0_STATUS(t0)
andi t0, KSU_USER | ST0_ERL | ST0_EXL
xori t0, KSU_USER
bnez t0, 1f /* If kernel */
INT_ADDIU t1, k1, VCPU_GUEST_KERNEL_ASID /* (BD) */
INT_ADDIU t1, k1, VCPU_GUEST_USER_ASID /* else user */
1:
/* t1: contains the base of the ASID array, need to get the cpu id */
LONG_L t2, TI_CPU($28) /* smp_processor_id */
INT_SLL t2, t2, 2 /* x4 */
REG_ADDU t3, t1, t2
LONG_L k0, (t3)
#ifdef CONFIG_MIPS_ASID_BITS_VARIABLE
li t3, CPUINFO_SIZE/4
mul t2, t2, t3 /* x sizeof(struct cpuinfo_mips)/4 */
LONG_L t2, (cpu_data + CPUINFO_ASID_MASK)(t2)
and k0, k0, t2
#else
andi k0, k0, MIPS_ENTRYHI_ASID
#endif
mtc0 k0, CP0_ENTRYHI
ehb
/* Disable RDHWR access */
mtc0 zero, CP0_HWRENA
.set noat
/* load the guest context from VCPU and return */
LONG_L $0, VCPU_R0(k1)
LONG_L $1, VCPU_R1(k1)
LONG_L $2, VCPU_R2(k1)
LONG_L $3, VCPU_R3(k1)
LONG_L $4, VCPU_R4(k1)
LONG_L $5, VCPU_R5(k1)
LONG_L $6, VCPU_R6(k1)
LONG_L $7, VCPU_R7(k1)
LONG_L $8, VCPU_R8(k1)
LONG_L $9, VCPU_R9(k1)
LONG_L $10, VCPU_R10(k1)
LONG_L $11, VCPU_R11(k1)
LONG_L $12, VCPU_R12(k1)
LONG_L $13, VCPU_R13(k1)
LONG_L $14, VCPU_R14(k1)
LONG_L $15, VCPU_R15(k1)
LONG_L $16, VCPU_R16(k1)
LONG_L $17, VCPU_R17(k1)
LONG_L $18, VCPU_R18(k1)
LONG_L $19, VCPU_R19(k1)
LONG_L $20, VCPU_R20(k1)
LONG_L $21, VCPU_R21(k1)
LONG_L $22, VCPU_R22(k1)
LONG_L $23, VCPU_R23(k1)
LONG_L $24, VCPU_R24(k1)
LONG_L $25, VCPU_R25(k1)
/* $/k1 loaded later */
LONG_L $28, VCPU_R28(k1)
LONG_L $29, VCPU_R29(k1)
LONG_L $30, VCPU_R30(k1)
LONG_L $31, VCPU_R31(k1)
FEXPORT(__kvm_mips_skip_guest_restore)
LONG_L k0, VCPU_HI(k1)
mthi k0
LONG_L k0, VCPU_LO(k1)
mtlo k0
LONG_L k0, VCPU_R26(k1)
LONG_L k1, VCPU_R27(k1)
eret
.set at
__kvm_mips_return_to_host:
/* EBASE is already pointing to Linux */
LONG_L k1, VCPU_HOST_STACK(k1)
INT_ADDIU k1,k1, -PT_SIZE
/* Restore host DDATA_LO */
LONG_L k0, PT_HOST_USERLOCAL(k1)
mtc0 k0, CP0_DDATA_LO
/*
* r2/v0 is the return code, shift it down by 2 (arithmetic)
* to recover the err code
*/
INT_SRA k0, v0, 2
move $2, k0
/* Load context saved on the host stack */
LONG_L $16, PT_R16(k1)
LONG_L $17, PT_R17(k1)
LONG_L $18, PT_R18(k1)
LONG_L $19, PT_R19(k1)
LONG_L $20, PT_R20(k1)
LONG_L $21, PT_R21(k1)
LONG_L $22, PT_R22(k1)
LONG_L $23, PT_R23(k1)
LONG_L $28, PT_R28(k1)
LONG_L $29, PT_R29(k1)
LONG_L $30, PT_R30(k1)
LONG_L k0, PT_HI(k1)
mthi k0
LONG_L k0, PT_LO(k1)
mtlo k0
/* Restore RDHWR access */
PTR_LI k0, 0x2000000F
mtc0 k0, CP0_HWRENA
/* Restore RA, which is the address we will return to */
LONG_L ra, PT_R31(k1)
j ra
nop
VECTOR_END(MIPSX(GuestExceptionEnd))
.end MIPSX(GuestException)
MIPSX(exceptions):
####
##### The exception handlers.
#####
.word _C_LABEL(MIPSX(GuestException)) # 0
.word _C_LABEL(MIPSX(GuestException)) # 1
.word _C_LABEL(MIPSX(GuestException)) # 2
.word _C_LABEL(MIPSX(GuestException)) # 3
.word _C_LABEL(MIPSX(GuestException)) # 4
.word _C_LABEL(MIPSX(GuestException)) # 5
.word _C_LABEL(MIPSX(GuestException)) # 6
.word _C_LABEL(MIPSX(GuestException)) # 7
.word _C_LABEL(MIPSX(GuestException)) # 8
.word _C_LABEL(MIPSX(GuestException)) # 9
.word _C_LABEL(MIPSX(GuestException)) # 10
.word _C_LABEL(MIPSX(GuestException)) # 11
.word _C_LABEL(MIPSX(GuestException)) # 12
.word _C_LABEL(MIPSX(GuestException)) # 13
.word _C_LABEL(MIPSX(GuestException)) # 14
.word _C_LABEL(MIPSX(GuestException)) # 15
.word _C_LABEL(MIPSX(GuestException)) # 16
.word _C_LABEL(MIPSX(GuestException)) # 17
.word _C_LABEL(MIPSX(GuestException)) # 18
.word _C_LABEL(MIPSX(GuestException)) # 19
.word _C_LABEL(MIPSX(GuestException)) # 20
.word _C_LABEL(MIPSX(GuestException)) # 21
.word _C_LABEL(MIPSX(GuestException)) # 22
.word _C_LABEL(MIPSX(GuestException)) # 23
.word _C_LABEL(MIPSX(GuestException)) # 24
.word _C_LABEL(MIPSX(GuestException)) # 25
.word _C_LABEL(MIPSX(GuestException)) # 26
.word _C_LABEL(MIPSX(GuestException)) # 27
.word _C_LABEL(MIPSX(GuestException)) # 28
.word _C_LABEL(MIPSX(GuestException)) # 29
.word _C_LABEL(MIPSX(GuestException)) # 30
.word _C_LABEL(MIPSX(GuestException)) # 31

367
arch/mips/kvm/mips.c
View File

@ -9,6 +9,7 @@
* Authors: Sanjay Lal <sanjayl@kymasys.com>
*/
#include <linux/bitops.h>
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/kdebug.h>
@ -147,7 +148,7 @@ void kvm_mips_free_vcpus(struct kvm *kvm)
/* Put the pages we reserved for the guest pmap */
for (i = 0; i < kvm->arch.guest_pmap_npages; i++) {
if (kvm->arch.guest_pmap[i] != KVM_INVALID_PAGE)
kvm_mips_release_pfn_clean(kvm->arch.guest_pmap[i]);
kvm_release_pfn_clean(kvm->arch.guest_pmap[i]);
}
kfree(kvm->arch.guest_pmap);
@ -244,10 +245,27 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
}
}
static inline void dump_handler(const char *symbol, void *start, void *end)
{
u32 *p;
pr_debug("LEAF(%s)\n", symbol);
pr_debug("\t.set push\n");
pr_debug("\t.set noreorder\n");
for (p = start; p < (u32 *)end; ++p)
pr_debug("\t.word\t0x%08x\t\t# %p\n", *p, p);
pr_debug("\t.set\tpop\n");
pr_debug("\tEND(%s)\n", symbol);
}
struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
{
int err, size, offset;
void *gebase;
int err, size;
void *gebase, *p, *handler;
int i;
struct kvm_vcpu *vcpu = kzalloc(sizeof(struct kvm_vcpu), GFP_KERNEL);
@ -273,9 +291,6 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
else
size = 0x4000;
/* Save Linux EBASE */
vcpu->arch.host_ebase = (void *)read_c0_ebase();
gebase = kzalloc(ALIGN(size, PAGE_SIZE), GFP_KERNEL);
if (!gebase) {
@ -285,44 +300,53 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
kvm_debug("Allocated %d bytes for KVM Exception Handlers @ %p\n",
ALIGN(size, PAGE_SIZE), gebase);
/*
* Check new ebase actually fits in CP0_EBase. The lack of a write gate
* limits us to the low 512MB of physical address space. If the memory
* we allocate is out of range, just give up now.
*/
if (!cpu_has_ebase_wg && virt_to_phys(gebase) >= 0x20000000) {
kvm_err("CP0_EBase.WG required for guest exception base %pK\n",
gebase);
err = -ENOMEM;
goto out_free_gebase;
}
/* Save new ebase */
vcpu->arch.guest_ebase = gebase;
/* Copy L1 Guest Exception handler to correct offset */
/* Build guest exception vectors dynamically in unmapped memory */
handler = gebase + 0x2000;
/* TLB Refill, EXL = 0 */
memcpy(gebase, mips32_exception,
mips32_exceptionEnd - mips32_exception);
kvm_mips_build_exception(gebase, handler);
/* General Exception Entry point */
memcpy(gebase + 0x180, mips32_exception,
mips32_exceptionEnd - mips32_exception);
kvm_mips_build_exception(gebase + 0x180, handler);
/* For vectored interrupts poke the exception code @ all offsets 0-7 */
for (i = 0; i < 8; i++) {
kvm_debug("L1 Vectored handler @ %p\n",
gebase + 0x200 + (i * VECTORSPACING));
memcpy(gebase + 0x200 + (i * VECTORSPACING), mips32_exception,
mips32_exceptionEnd - mips32_exception);
kvm_mips_build_exception(gebase + 0x200 + i * VECTORSPACING,
handler);
}
/* General handler, relocate to unmapped space for sanity's sake */
offset = 0x2000;
kvm_debug("Installing KVM Exception handlers @ %p, %#x bytes\n",
gebase + offset,
mips32_GuestExceptionEnd - mips32_GuestException);
/* General exit handler */
p = handler;
p = kvm_mips_build_exit(p);
memcpy(gebase + offset, mips32_GuestException,
mips32_GuestExceptionEnd - mips32_GuestException);
/* Guest entry routine */
vcpu->arch.vcpu_run = p;
p = kvm_mips_build_vcpu_run(p);
#ifdef MODULE
offset += mips32_GuestExceptionEnd - mips32_GuestException;
memcpy(gebase + offset, (char *)__kvm_mips_vcpu_run,
__kvm_mips_vcpu_run_end - (char *)__kvm_mips_vcpu_run);
vcpu->arch.vcpu_run = gebase + offset;
#else
vcpu->arch.vcpu_run = __kvm_mips_vcpu_run;
#endif
/* Dump the generated code */
pr_debug("#include <asm/asm.h>\n");
pr_debug("#include <asm/regdef.h>\n");
pr_debug("\n");
dump_handler("kvm_vcpu_run", vcpu->arch.vcpu_run, p);
dump_handler("kvm_gen_exc", gebase + 0x180, gebase + 0x200);
dump_handler("kvm_exit", gebase + 0x2000, vcpu->arch.vcpu_run);
/* Invalidate the icache for these ranges */
local_flush_icache_range((unsigned long)gebase,
@ -408,17 +432,19 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
kvm_mips_deliver_interrupts(vcpu,
kvm_read_c0_guest_cause(vcpu->arch.cop0));
__kvm_guest_enter();
guest_enter_irqoff();
/* Disable hardware page table walking while in guest */
htw_stop();
trace_kvm_enter(vcpu);
r = vcpu->arch.vcpu_run(run, vcpu);
trace_kvm_out(vcpu);
/* Re-enable HTW before enabling interrupts */
htw_start();
__kvm_guest_exit();
guest_exit_irqoff();
local_irq_enable();
if (vcpu->sigset_active)
@ -507,8 +533,10 @@ static u64 kvm_mips_get_one_regs[] = {
KVM_REG_MIPS_R30,
KVM_REG_MIPS_R31,
#ifndef CONFIG_CPU_MIPSR6
KVM_REG_MIPS_HI,
KVM_REG_MIPS_LO,
#endif
KVM_REG_MIPS_PC,
KVM_REG_MIPS_CP0_INDEX,
@ -539,6 +567,104 @@ static u64 kvm_mips_get_one_regs[] = {
KVM_REG_MIPS_COUNT_HZ,
};
static u64 kvm_mips_get_one_regs_fpu[] = {
KVM_REG_MIPS_FCR_IR,
KVM_REG_MIPS_FCR_CSR,
};
static u64 kvm_mips_get_one_regs_msa[] = {
KVM_REG_MIPS_MSA_IR,
KVM_REG_MIPS_MSA_CSR,
};
static u64 kvm_mips_get_one_regs_kscratch[] = {
KVM_REG_MIPS_CP0_KSCRATCH1,
KVM_REG_MIPS_CP0_KSCRATCH2,
KVM_REG_MIPS_CP0_KSCRATCH3,
KVM_REG_MIPS_CP0_KSCRATCH4,
KVM_REG_MIPS_CP0_KSCRATCH5,
KVM_REG_MIPS_CP0_KSCRATCH6,
};
static unsigned long kvm_mips_num_regs(struct kvm_vcpu *vcpu)
{
unsigned long ret;
ret = ARRAY_SIZE(kvm_mips_get_one_regs);
if (kvm_mips_guest_can_have_fpu(&vcpu->arch)) {
ret += ARRAY_SIZE(kvm_mips_get_one_regs_fpu) + 48;
/* odd doubles */
if (boot_cpu_data.fpu_id & MIPS_FPIR_F64)
ret += 16;
}
if (kvm_mips_guest_can_have_msa(&vcpu->arch))
ret += ARRAY_SIZE(kvm_mips_get_one_regs_msa) + 32;
ret += __arch_hweight8(vcpu->arch.kscratch_enabled);
ret += kvm_mips_callbacks->num_regs(vcpu);
return ret;
}
static int kvm_mips_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices)
{
u64 index;
unsigned int i;
if (copy_to_user(indices, kvm_mips_get_one_regs,
sizeof(kvm_mips_get_one_regs)))
return -EFAULT;
indices += ARRAY_SIZE(kvm_mips_get_one_regs);
if (kvm_mips_guest_can_have_fpu(&vcpu->arch)) {
if (copy_to_user(indices, kvm_mips_get_one_regs_fpu,
sizeof(kvm_mips_get_one_regs_fpu)))
return -EFAULT;
indices += ARRAY_SIZE(kvm_mips_get_one_regs_fpu);
for (i = 0; i < 32; ++i) {
index = KVM_REG_MIPS_FPR_32(i);
if (copy_to_user(indices, &index, sizeof(index)))
return -EFAULT;
++indices;
/* skip odd doubles if no F64 */
if (i & 1 && !(boot_cpu_data.fpu_id & MIPS_FPIR_F64))
continue;
index = KVM_REG_MIPS_FPR_64(i);
if (copy_to_user(indices, &index, sizeof(index)))
return -EFAULT;
++indices;
}
}
if (kvm_mips_guest_can_have_msa(&vcpu->arch)) {
if (copy_to_user(indices, kvm_mips_get_one_regs_msa,
sizeof(kvm_mips_get_one_regs_msa)))
return -EFAULT;
indices += ARRAY_SIZE(kvm_mips_get_one_regs_msa);
for (i = 0; i < 32; ++i) {
index = KVM_REG_MIPS_VEC_128(i);
if (copy_to_user(indices, &index, sizeof(index)))
return -EFAULT;
++indices;
}
}
for (i = 0; i < 6; ++i) {
if (!(vcpu->arch.kscratch_enabled & BIT(i + 2)))
continue;
if (copy_to_user(indices, &kvm_mips_get_one_regs_kscratch[i],
sizeof(kvm_mips_get_one_regs_kscratch[i])))
return -EFAULT;
++indices;
}
return kvm_mips_callbacks->copy_reg_indices(vcpu, indices);
}
static int kvm_mips_get_reg(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg)
{
@ -554,12 +680,14 @@ static int kvm_mips_get_reg(struct kvm_vcpu *vcpu,
case KVM_REG_MIPS_R0 ... KVM_REG_MIPS_R31:
v = (long)vcpu->arch.gprs[reg->id - KVM_REG_MIPS_R0];
break;
#ifndef CONFIG_CPU_MIPSR6
case KVM_REG_MIPS_HI:
v = (long)vcpu->arch.hi;
break;
case KVM_REG_MIPS_LO:
v = (long)vcpu->arch.lo;
break;
#endif
case KVM_REG_MIPS_PC:
v = (long)vcpu->arch.pc;
break;
@ -688,17 +816,37 @@ static int kvm_mips_get_reg(struct kvm_vcpu *vcpu,
case KVM_REG_MIPS_CP0_ERROREPC:
v = (long)kvm_read_c0_guest_errorepc(cop0);
break;
case KVM_REG_MIPS_CP0_KSCRATCH1 ... KVM_REG_MIPS_CP0_KSCRATCH6:
idx = reg->id - KVM_REG_MIPS_CP0_KSCRATCH1 + 2;
if (!(vcpu->arch.kscratch_enabled & BIT(idx)))
return -EINVAL;
switch (idx) {
case 2:
v = (long)kvm_read_c0_guest_kscratch1(cop0);
break;
case 3:
v = (long)kvm_read_c0_guest_kscratch2(cop0);
break;
case 4:
v = (long)kvm_read_c0_guest_kscratch3(cop0);
break;
case 5:
v = (long)kvm_read_c0_guest_kscratch4(cop0);
break;
case 6:
v = (long)kvm_read_c0_guest_kscratch5(cop0);
break;
case 7:
v = (long)kvm_read_c0_guest_kscratch6(cop0);
break;
}
break;
/* registers to be handled specially */
case KVM_REG_MIPS_CP0_COUNT:
case KVM_REG_MIPS_COUNT_CTL:
case KVM_REG_MIPS_COUNT_RESUME:
case KVM_REG_MIPS_COUNT_HZ:
default:
ret = kvm_mips_callbacks->get_one_reg(vcpu, reg, &v);
if (ret)
return ret;
break;
default:
return -EINVAL;
}
if ((reg->id & KVM_REG_SIZE_MASK) == KVM_REG_SIZE_U64) {
u64 __user *uaddr64 = (u64 __user *)(long)reg->addr;
@ -755,12 +903,14 @@ static int kvm_mips_set_reg(struct kvm_vcpu *vcpu,
case KVM_REG_MIPS_R1 ... KVM_REG_MIPS_R31:
vcpu->arch.gprs[reg->id - KVM_REG_MIPS_R0] = v;
break;
#ifndef CONFIG_CPU_MIPSR6
case KVM_REG_MIPS_HI:
vcpu->arch.hi = v;
break;
case KVM_REG_MIPS_LO:
vcpu->arch.lo = v;
break;
#endif
case KVM_REG_MIPS_PC:
vcpu->arch.pc = v;
break;
@ -859,22 +1009,34 @@ static int kvm_mips_set_reg(struct kvm_vcpu *vcpu,
case KVM_REG_MIPS_CP0_ERROREPC:
kvm_write_c0_guest_errorepc(cop0, v);
break;
case KVM_REG_MIPS_CP0_KSCRATCH1 ... KVM_REG_MIPS_CP0_KSCRATCH6:
idx = reg->id - KVM_REG_MIPS_CP0_KSCRATCH1 + 2;
if (!(vcpu->arch.kscratch_enabled & BIT(idx)))
return -EINVAL;
switch (idx) {
case 2:
kvm_write_c0_guest_kscratch1(cop0, v);
break;
case 3:
kvm_write_c0_guest_kscratch2(cop0, v);
break;
case 4:
kvm_write_c0_guest_kscratch3(cop0, v);
break;
case 5:
kvm_write_c0_guest_kscratch4(cop0, v);
break;
case 6:
kvm_write_c0_guest_kscratch5(cop0, v);
break;
case 7:
kvm_write_c0_guest_kscratch6(cop0, v);
break;
}
break;
/* registers to be handled specially */
case KVM_REG_MIPS_CP0_COUNT:
case KVM_REG_MIPS_CP0_COMPARE:
case KVM_REG_MIPS_CP0_CAUSE:
case KVM_REG_MIPS_CP0_CONFIG:
case KVM_REG_MIPS_CP0_CONFIG1:
case KVM_REG_MIPS_CP0_CONFIG2:
case KVM_REG_MIPS_CP0_CONFIG3:
case KVM_REG_MIPS_CP0_CONFIG4:
case KVM_REG_MIPS_CP0_CONFIG5:
case KVM_REG_MIPS_COUNT_CTL:
case KVM_REG_MIPS_COUNT_RESUME:
case KVM_REG_MIPS_COUNT_HZ:
return kvm_mips_callbacks->set_one_reg(vcpu, reg, v);
default:
return -EINVAL;
return kvm_mips_callbacks->set_one_reg(vcpu, reg, v);
}
return 0;
}
@ -927,23 +1089,18 @@ long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl,
}
case KVM_GET_REG_LIST: {
struct kvm_reg_list __user *user_list = argp;
u64 __user *reg_dest;
struct kvm_reg_list reg_list;
unsigned n;
if (copy_from_user(&reg_list, user_list, sizeof(reg_list)))
return -EFAULT;
n = reg_list.n;
reg_list.n = ARRAY_SIZE(kvm_mips_get_one_regs);
reg_list.n = kvm_mips_num_regs(vcpu);
if (copy_to_user(user_list, &reg_list, sizeof(reg_list)))
return -EFAULT;
if (n < reg_list.n)
return -E2BIG;
reg_dest = user_list->reg;
if (copy_to_user(reg_dest, kvm_mips_get_one_regs,
sizeof(kvm_mips_get_one_regs)))
return -EFAULT;
return 0;
return kvm_mips_copy_reg_indices(vcpu, user_list->reg);
}
case KVM_NMI:
/* Treat the NMI as a CPU reset */
@ -1222,7 +1379,7 @@ int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu)
static void kvm_mips_set_c0_status(void)
{
uint32_t status = read_c0_status();
u32 status = read_c0_status();
if (cpu_has_dsp)
status |= (ST0_MX);
@ -1236,9 +1393,9 @@ static void kvm_mips_set_c0_status(void)
*/
int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
{
uint32_t cause = vcpu->arch.host_cp0_cause;
uint32_t exccode = (cause >> CAUSEB_EXCCODE) & 0x1f;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
u32 exccode = (cause >> CAUSEB_EXCCODE) & 0x1f;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
unsigned long badvaddr = vcpu->arch.host_cp0_badvaddr;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -1260,6 +1417,7 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
kvm_debug("kvm_mips_handle_exit: cause: %#x, PC: %p, kvm_run: %p, kvm_vcpu: %p\n",
cause, opc, run, vcpu);
trace_kvm_exit(vcpu, exccode);
/*
* Do a privilege check, if in UM most of these exit conditions end up
@ -1279,7 +1437,6 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
kvm_debug("[%d]EXCCODE_INT @ %p\n", vcpu->vcpu_id, opc);
++vcpu->stat.int_exits;
trace_kvm_exit(vcpu, INT_EXITS);
if (need_resched())
cond_resched();
@ -1291,7 +1448,6 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
kvm_debug("EXCCODE_CPU: @ PC: %p\n", opc);
++vcpu->stat.cop_unusable_exits;
trace_kvm_exit(vcpu, COP_UNUSABLE_EXITS);
ret = kvm_mips_callbacks->handle_cop_unusable(vcpu);
/* XXXKYMA: Might need to return to user space */
if (run->exit_reason == KVM_EXIT_IRQ_WINDOW_OPEN)
@ -1300,7 +1456,6 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
case EXCCODE_MOD:
++vcpu->stat.tlbmod_exits;
trace_kvm_exit(vcpu, TLBMOD_EXITS);
ret = kvm_mips_callbacks->handle_tlb_mod(vcpu);
break;
@ -1310,7 +1465,6 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
badvaddr);
++vcpu->stat.tlbmiss_st_exits;
trace_kvm_exit(vcpu, TLBMISS_ST_EXITS);
ret = kvm_mips_callbacks->handle_tlb_st_miss(vcpu);
break;
@ -1319,61 +1473,51 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
cause, opc, badvaddr);
++vcpu->stat.tlbmiss_ld_exits;
trace_kvm_exit(vcpu, TLBMISS_LD_EXITS);
ret = kvm_mips_callbacks->handle_tlb_ld_miss(vcpu);
break;
case EXCCODE_ADES:
++vcpu->stat.addrerr_st_exits;
trace_kvm_exit(vcpu, ADDRERR_ST_EXITS);
ret = kvm_mips_callbacks->handle_addr_err_st(vcpu);
break;
case EXCCODE_ADEL:
++vcpu->stat.addrerr_ld_exits;
trace_kvm_exit(vcpu, ADDRERR_LD_EXITS);
ret = kvm_mips_callbacks->handle_addr_err_ld(vcpu);
break;
case EXCCODE_SYS:
++vcpu->stat.syscall_exits;
trace_kvm_exit(vcpu, SYSCALL_EXITS);
ret = kvm_mips_callbacks->handle_syscall(vcpu);
break;
case EXCCODE_RI:
++vcpu->stat.resvd_inst_exits;
trace_kvm_exit(vcpu, RESVD_INST_EXITS);
ret = kvm_mips_callbacks->handle_res_inst(vcpu);
break;
case EXCCODE_BP:
++vcpu->stat.break_inst_exits;
trace_kvm_exit(vcpu, BREAK_INST_EXITS);
ret = kvm_mips_callbacks->handle_break(vcpu);
break;
case EXCCODE_TR:
++vcpu->stat.trap_inst_exits;
trace_kvm_exit(vcpu, TRAP_INST_EXITS);
ret = kvm_mips_callbacks->handle_trap(vcpu);
break;
case EXCCODE_MSAFPE:
++vcpu->stat.msa_fpe_exits;
trace_kvm_exit(vcpu, MSA_FPE_EXITS);
ret = kvm_mips_callbacks->handle_msa_fpe(vcpu);
break;
case EXCCODE_FPE:
++vcpu->stat.fpe_exits;
trace_kvm_exit(vcpu, FPE_EXITS);
ret = kvm_mips_callbacks->handle_fpe(vcpu);
break;
case EXCCODE_MSADIS:
++vcpu->stat.msa_disabled_exits;
trace_kvm_exit(vcpu, MSA_DISABLED_EXITS);
ret = kvm_mips_callbacks->handle_msa_disabled(vcpu);
break;
@ -1400,11 +1544,13 @@ int kvm_mips_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu)
run->exit_reason = KVM_EXIT_INTR;
ret = (-EINTR << 2) | RESUME_HOST;
++vcpu->stat.signal_exits;
trace_kvm_exit(vcpu, SIGNAL_EXITS);
trace_kvm_exit(vcpu, KVM_TRACE_EXIT_SIGNAL);
}
}
if (ret == RESUME_GUEST) {
trace_kvm_reenter(vcpu);
/*
* If FPU / MSA are enabled (i.e. the guest's FPU / MSA context
* is live), restore FCR31 / MSACSR.
@ -1450,7 +1596,7 @@ void kvm_own_fpu(struct kvm_vcpu *vcpu)
* not to clobber the status register directly via the commpage.
*/
if (cpu_has_msa && sr & ST0_CU1 && !(sr & ST0_FR) &&
vcpu->arch.fpu_inuse & KVM_MIPS_FPU_MSA)
vcpu->arch.aux_inuse & KVM_MIPS_AUX_MSA)
kvm_lose_fpu(vcpu);
/*
@ -1465,9 +1611,12 @@ void kvm_own_fpu(struct kvm_vcpu *vcpu)
enable_fpu_hazard();
/* If guest FPU state not active, restore it now */
if (!(vcpu->arch.fpu_inuse & KVM_MIPS_FPU_FPU)) {
if (!(vcpu->arch.aux_inuse & KVM_MIPS_AUX_FPU)) {
__kvm_restore_fpu(&vcpu->arch);
vcpu->arch.fpu_inuse |= KVM_MIPS_FPU_FPU;
vcpu->arch.aux_inuse |= KVM_MIPS_AUX_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_FPU);
} else {
trace_kvm_aux(vcpu, KVM_TRACE_AUX_ENABLE, KVM_TRACE_AUX_FPU);
}
preempt_enable();
@ -1494,8 +1643,8 @@ void kvm_own_msa(struct kvm_vcpu *vcpu)
* interacts with MSA state, so play it safe and save it first.
*/
if (!(sr & ST0_FR) &&
(vcpu->arch.fpu_inuse & (KVM_MIPS_FPU_FPU |
KVM_MIPS_FPU_MSA)) == KVM_MIPS_FPU_FPU)
(vcpu->arch.aux_inuse & (KVM_MIPS_AUX_FPU |
KVM_MIPS_AUX_MSA)) == KVM_MIPS_AUX_FPU)
kvm_lose_fpu(vcpu);
change_c0_status(ST0_CU1 | ST0_FR, sr);
@ -1509,22 +1658,26 @@ void kvm_own_msa(struct kvm_vcpu *vcpu)
set_c0_config5(MIPS_CONF5_MSAEN);
enable_fpu_hazard();
switch (vcpu->arch.fpu_inuse & (KVM_MIPS_FPU_FPU | KVM_MIPS_FPU_MSA)) {
case KVM_MIPS_FPU_FPU:
switch (vcpu->arch.aux_inuse & (KVM_MIPS_AUX_FPU | KVM_MIPS_AUX_MSA)) {
case KVM_MIPS_AUX_FPU:
/*
* Guest FPU state already loaded, only restore upper MSA state
*/
__kvm_restore_msa_upper(&vcpu->arch);
vcpu->arch.fpu_inuse |= KVM_MIPS_FPU_MSA;
vcpu->arch.aux_inuse |= KVM_MIPS_AUX_MSA;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE, KVM_TRACE_AUX_MSA);
break;
case 0:
/* Neither FPU or MSA already active, restore full MSA state */
__kvm_restore_msa(&vcpu->arch);
vcpu->arch.fpu_inuse |= KVM_MIPS_FPU_MSA;
vcpu->arch.aux_inuse |= KVM_MIPS_AUX_MSA;
if (kvm_mips_guest_has_fpu(&vcpu->arch))
vcpu->arch.fpu_inuse |= KVM_MIPS_FPU_FPU;
vcpu->arch.aux_inuse |= KVM_MIPS_AUX_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_RESTORE,
KVM_TRACE_AUX_FPU_MSA);
break;
default:
trace_kvm_aux(vcpu, KVM_TRACE_AUX_ENABLE, KVM_TRACE_AUX_MSA);
break;
}
@ -1536,13 +1689,15 @@ void kvm_own_msa(struct kvm_vcpu *vcpu)
void kvm_drop_fpu(struct kvm_vcpu *vcpu)
{
preempt_disable();
if (cpu_has_msa && vcpu->arch.fpu_inuse & KVM_MIPS_FPU_MSA) {
if (cpu_has_msa && vcpu->arch.aux_inuse & KVM_MIPS_AUX_MSA) {
disable_msa();
vcpu->arch.fpu_inuse &= ~KVM_MIPS_FPU_MSA;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_DISCARD, KVM_TRACE_AUX_MSA);
vcpu->arch.aux_inuse &= ~KVM_MIPS_AUX_MSA;
}
if (vcpu->arch.fpu_inuse & KVM_MIPS_FPU_FPU) {
if (vcpu->arch.aux_inuse & KVM_MIPS_AUX_FPU) {
clear_c0_status(ST0_CU1 | ST0_FR);
vcpu->arch.fpu_inuse &= ~KVM_MIPS_FPU_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_DISCARD, KVM_TRACE_AUX_FPU);
vcpu->arch.aux_inuse &= ~KVM_MIPS_AUX_FPU;
}
preempt_enable();
}
@ -1558,25 +1713,27 @@ void kvm_lose_fpu(struct kvm_vcpu *vcpu)
*/
preempt_disable();
if (cpu_has_msa && vcpu->arch.fpu_inuse & KVM_MIPS_FPU_MSA) {
if (cpu_has_msa && vcpu->arch.aux_inuse & KVM_MIPS_AUX_MSA) {
set_c0_config5(MIPS_CONF5_MSAEN);
enable_fpu_hazard();
__kvm_save_msa(&vcpu->arch);
trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_FPU_MSA);
/* Disable MSA & FPU */
disable_msa();
if (vcpu->arch.fpu_inuse & KVM_MIPS_FPU_FPU) {
if (vcpu->arch.aux_inuse & KVM_MIPS_AUX_FPU) {
clear_c0_status(ST0_CU1 | ST0_FR);
disable_fpu_hazard();
}
vcpu->arch.fpu_inuse &= ~(KVM_MIPS_FPU_FPU | KVM_MIPS_FPU_MSA);
} else if (vcpu->arch.fpu_inuse & KVM_MIPS_FPU_FPU) {
vcpu->arch.aux_inuse &= ~(KVM_MIPS_AUX_FPU | KVM_MIPS_AUX_MSA);
} else if (vcpu->arch.aux_inuse & KVM_MIPS_AUX_FPU) {
set_c0_status(ST0_CU1);
enable_fpu_hazard();
__kvm_save_fpu(&vcpu->arch);
vcpu->arch.fpu_inuse &= ~KVM_MIPS_FPU_FPU;
vcpu->arch.aux_inuse &= ~KVM_MIPS_AUX_FPU;
trace_kvm_aux(vcpu, KVM_TRACE_AUX_SAVE, KVM_TRACE_AUX_FPU);
/* Disable FPU */
clear_c0_status(ST0_CU1 | ST0_FR);
@ -1638,6 +1795,10 @@ static int __init kvm_mips_init(void)
{
int ret;
ret = kvm_mips_entry_setup();
if (ret)
return ret;
ret = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
if (ret)
@ -1645,18 +1806,6 @@ static int __init kvm_mips_init(void)
register_die_notifier(&kvm_mips_csr_die_notifier);
/*
* On MIPS, kernel modules are executed from "mapped space", which
* requires TLBs. The TLB handling code is statically linked with
* the rest of the kernel (tlb.c) to avoid the possibility of
* double faulting. The issue is that the TLB code references
* routines that are part of the the KVM module, which are only
* available once the module is loaded.
*/
kvm_mips_gfn_to_pfn = gfn_to_pfn;
kvm_mips_release_pfn_clean = kvm_release_pfn_clean;
kvm_mips_is_error_pfn = is_error_pfn;
return 0;
}
@ -1664,10 +1813,6 @@ static void __exit kvm_mips_exit(void)
{
kvm_exit();
kvm_mips_gfn_to_pfn = NULL;
kvm_mips_release_pfn_clean = NULL;
kvm_mips_is_error_pfn = NULL;
unregister_die_notifier(&kvm_mips_csr_die_notifier);
}

375
arch/mips/kvm/mmu.c Normal file
View File

@ -0,0 +1,375 @@
/*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file "COPYING" in the main directory of this archive
* for more details.
*
* KVM/MIPS MMU handling in the KVM module.
*
* Copyright (C) 2012 MIPS Technologies, Inc. All rights reserved.
* Authors: Sanjay Lal <sanjayl@kymasys.com>
*/
#include <linux/highmem.h>
#include <linux/kvm_host.h>
#include <asm/mmu_context.h>
static u32 kvm_mips_get_kernel_asid(struct kvm_vcpu *vcpu)
{
int cpu = smp_processor_id();
return vcpu->arch.guest_kernel_asid[cpu] &
cpu_asid_mask(&cpu_data[cpu]);
}
static u32 kvm_mips_get_user_asid(struct kvm_vcpu *vcpu)
{
int cpu = smp_processor_id();
return vcpu->arch.guest_user_asid[cpu] &
cpu_asid_mask(&cpu_data[cpu]);
}
static int kvm_mips_map_page(struct kvm *kvm, gfn_t gfn)
{
int srcu_idx, err = 0;
kvm_pfn_t pfn;
if (kvm->arch.guest_pmap[gfn] != KVM_INVALID_PAGE)
return 0;
srcu_idx = srcu_read_lock(&kvm->srcu);
pfn = gfn_to_pfn(kvm, gfn);
if (is_error_pfn(pfn)) {
kvm_err("Couldn't get pfn for gfn %#llx!\n", gfn);
err = -EFAULT;
goto out;
}
kvm->arch.guest_pmap[gfn] = pfn;
out:
srcu_read_unlock(&kvm->srcu, srcu_idx);
return err;
}
/* Translate guest KSEG0 addresses to Host PA */
unsigned long kvm_mips_translate_guest_kseg0_to_hpa(struct kvm_vcpu *vcpu,
unsigned long gva)
{
gfn_t gfn;
unsigned long offset = gva & ~PAGE_MASK;
struct kvm *kvm = vcpu->kvm;
if (KVM_GUEST_KSEGX(gva) != KVM_GUEST_KSEG0) {
kvm_err("%s/%p: Invalid gva: %#lx\n", __func__,
__builtin_return_address(0), gva);
return KVM_INVALID_PAGE;
}
gfn = (KVM_GUEST_CPHYSADDR(gva) >> PAGE_SHIFT);
if (gfn >= kvm->arch.guest_pmap_npages) {
kvm_err("%s: Invalid gfn: %#llx, GVA: %#lx\n", __func__, gfn,
gva);
return KVM_INVALID_PAGE;
}
if (kvm_mips_map_page(vcpu->kvm, gfn) < 0)
return KVM_INVALID_ADDR;
return (kvm->arch.guest_pmap[gfn] << PAGE_SHIFT) + offset;
}
/* XXXKYMA: Must be called with interrupts disabled */
int kvm_mips_handle_kseg0_tlb_fault(unsigned long badvaddr,
struct kvm_vcpu *vcpu)
{
gfn_t gfn;
kvm_pfn_t pfn0, pfn1;
unsigned long vaddr = 0;
unsigned long entryhi = 0, entrylo0 = 0, entrylo1 = 0;
struct kvm *kvm = vcpu->kvm;
const int flush_dcache_mask = 0;
int ret;
if (KVM_GUEST_KSEGX(badvaddr) != KVM_GUEST_KSEG0) {
kvm_err("%s: Invalid BadVaddr: %#lx\n", __func__, badvaddr);
kvm_mips_dump_host_tlbs();
return -1;
}
gfn = (KVM_GUEST_CPHYSADDR(badvaddr) >> PAGE_SHIFT);
if (gfn >= kvm->arch.guest_pmap_npages) {
kvm_err("%s: Invalid gfn: %#llx, BadVaddr: %#lx\n", __func__,
gfn, badvaddr);
kvm_mips_dump_host_tlbs();
return -1;
}
vaddr = badvaddr & (PAGE_MASK << 1);
if (kvm_mips_map_page(vcpu->kvm, gfn) < 0)
return -1;
if (kvm_mips_map_page(vcpu->kvm, gfn ^ 0x1) < 0)
return -1;
pfn0 = kvm->arch.guest_pmap[gfn & ~0x1];
pfn1 = kvm->arch.guest_pmap[gfn | 0x1];
entrylo0 = mips3_paddr_to_tlbpfn(pfn0 << PAGE_SHIFT) |
((_page_cachable_default >> _CACHE_SHIFT) << ENTRYLO_C_SHIFT) |
ENTRYLO_D | ENTRYLO_V;
entrylo1 = mips3_paddr_to_tlbpfn(pfn1 << PAGE_SHIFT) |
((_page_cachable_default >> _CACHE_SHIFT) << ENTRYLO_C_SHIFT) |
ENTRYLO_D | ENTRYLO_V;
preempt_disable();
entryhi = (vaddr | kvm_mips_get_kernel_asid(vcpu));
ret = kvm_mips_host_tlb_write(vcpu, entryhi, entrylo0, entrylo1,
flush_dcache_mask);
preempt_enable();
return ret;
}
int kvm_mips_handle_mapped_seg_tlb_fault(struct kvm_vcpu *vcpu,
struct kvm_mips_tlb *tlb)
{
unsigned long entryhi = 0, entrylo0 = 0, entrylo1 = 0;
struct kvm *kvm = vcpu->kvm;
kvm_pfn_t pfn0, pfn1;
int ret;
if ((tlb->tlb_hi & VPN2_MASK) == 0) {
pfn0 = 0;
pfn1 = 0;
} else {
if (kvm_mips_map_page(kvm, mips3_tlbpfn_to_paddr(tlb->tlb_lo[0])
>> PAGE_SHIFT) < 0)
return -1;
if (kvm_mips_map_page(kvm, mips3_tlbpfn_to_paddr(tlb->tlb_lo[1])
>> PAGE_SHIFT) < 0)
return -1;
pfn0 = kvm->arch.guest_pmap[
mips3_tlbpfn_to_paddr(tlb->tlb_lo[0]) >> PAGE_SHIFT];
pfn1 = kvm->arch.guest_pmap[
mips3_tlbpfn_to_paddr(tlb->tlb_lo[1]) >> PAGE_SHIFT];
}
/* Get attributes from the Guest TLB */
entrylo0 = mips3_paddr_to_tlbpfn(pfn0 << PAGE_SHIFT) |
((_page_cachable_default >> _CACHE_SHIFT) << ENTRYLO_C_SHIFT) |
(tlb->tlb_lo[0] & ENTRYLO_D) |
(tlb->tlb_lo[0] & ENTRYLO_V);
entrylo1 = mips3_paddr_to_tlbpfn(pfn1 << PAGE_SHIFT) |
((_page_cachable_default >> _CACHE_SHIFT) << ENTRYLO_C_SHIFT) |
(tlb->tlb_lo[1] & ENTRYLO_D) |
(tlb->tlb_lo[1] & ENTRYLO_V);
kvm_debug("@ %#lx tlb_lo0: 0x%08lx tlb_lo1: 0x%08lx\n", vcpu->arch.pc,
tlb->tlb_lo[0], tlb->tlb_lo[1]);
preempt_disable();
entryhi = (tlb->tlb_hi & VPN2_MASK) | (KVM_GUEST_KERNEL_MODE(vcpu) ?
kvm_mips_get_kernel_asid(vcpu) :
kvm_mips_get_user_asid(vcpu));
ret = kvm_mips_host_tlb_write(vcpu, entryhi, entrylo0, entrylo1,
tlb->tlb_mask);
preempt_enable();
return ret;
}
void kvm_get_new_mmu_context(struct mm_struct *mm, unsigned long cpu,
struct kvm_vcpu *vcpu)
{
unsigned long asid = asid_cache(cpu);
asid += cpu_asid_inc();
if (!(asid & cpu_asid_mask(&cpu_data[cpu]))) {
if (cpu_has_vtag_icache)
flush_icache_all();
kvm_local_flush_tlb_all(); /* start new asid cycle */
if (!asid) /* fix version if needed */
asid = asid_first_version(cpu);
}
cpu_context(cpu, mm) = asid_cache(cpu) = asid;
}
/**
* kvm_mips_migrate_count() - Migrate timer.
* @vcpu: Virtual CPU.
*
* Migrate CP0_Count hrtimer to the current CPU by cancelling and restarting it
* if it was running prior to being cancelled.
*
* Must be called when the VCPU is migrated to a different CPU to ensure that
* timer expiry during guest execution interrupts the guest and causes the
* interrupt to be delivered in a timely manner.
*/
static void kvm_mips_migrate_count(struct kvm_vcpu *vcpu)
{
if (hrtimer_cancel(&vcpu->arch.comparecount_timer))
hrtimer_restart(&vcpu->arch.comparecount_timer);
}
/* Restore ASID once we are scheduled back after preemption */
void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
{
unsigned long asid_mask = cpu_asid_mask(&cpu_data[cpu]);
unsigned long flags;
int newasid = 0;
kvm_debug("%s: vcpu %p, cpu: %d\n", __func__, vcpu, cpu);
/* Allocate new kernel and user ASIDs if needed */
local_irq_save(flags);
if ((vcpu->arch.guest_kernel_asid[cpu] ^ asid_cache(cpu)) &
asid_version_mask(cpu)) {
kvm_get_new_mmu_context(&vcpu->arch.guest_kernel_mm, cpu, vcpu);
vcpu->arch.guest_kernel_asid[cpu] =
vcpu->arch.guest_kernel_mm.context.asid[cpu];
kvm_get_new_mmu_context(&vcpu->arch.guest_user_mm, cpu, vcpu);
vcpu->arch.guest_user_asid[cpu] =
vcpu->arch.guest_user_mm.context.asid[cpu];
newasid++;
kvm_debug("[%d]: cpu_context: %#lx\n", cpu,
cpu_context(cpu, current->mm));
kvm_debug("[%d]: Allocated new ASID for Guest Kernel: %#x\n",
cpu, vcpu->arch.guest_kernel_asid[cpu]);
kvm_debug("[%d]: Allocated new ASID for Guest User: %#x\n", cpu,
vcpu->arch.guest_user_asid[cpu]);
}
if (vcpu->arch.last_sched_cpu != cpu) {
kvm_debug("[%d->%d]KVM VCPU[%d] switch\n",
vcpu->arch.last_sched_cpu, cpu, vcpu->vcpu_id);
/*
* Migrate the timer interrupt to the current CPU so that it
* always interrupts the guest and synchronously triggers a
* guest timer interrupt.
*/
kvm_mips_migrate_count(vcpu);
}
if (!newasid) {
/*
* If we preempted while the guest was executing, then reload
* the pre-empted ASID
*/
if (current->flags & PF_VCPU) {
write_c0_entryhi(vcpu->arch.
preempt_entryhi & asid_mask);
ehb();
}
} else {
/* New ASIDs were allocated for the VM */
/*
* Were we in guest context? If so then the pre-empted ASID is
* no longer valid, we need to set it to what it should be based
* on the mode of the Guest (Kernel/User)
*/
if (current->flags & PF_VCPU) {
if (KVM_GUEST_KERNEL_MODE(vcpu))
write_c0_entryhi(vcpu->arch.
guest_kernel_asid[cpu] &
asid_mask);
else
write_c0_entryhi(vcpu->arch.
guest_user_asid[cpu] &
asid_mask);
ehb();
}
}
/* restore guest state to registers */
kvm_mips_callbacks->vcpu_set_regs(vcpu);
local_irq_restore(flags);
}
/* ASID can change if another task is scheduled during preemption */
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
unsigned long flags;
int cpu;
local_irq_save(flags);
cpu = smp_processor_id();
vcpu->arch.preempt_entryhi = read_c0_entryhi();
vcpu->arch.last_sched_cpu = cpu;
/* save guest state in registers */
kvm_mips_callbacks->vcpu_get_regs(vcpu);
if (((cpu_context(cpu, current->mm) ^ asid_cache(cpu)) &
asid_version_mask(cpu))) {
kvm_debug("%s: Dropping MMU Context: %#lx\n", __func__,
cpu_context(cpu, current->mm));
drop_mmu_context(current->mm, cpu);
}
write_c0_entryhi(cpu_asid(cpu, current->mm));
ehb();
local_irq_restore(flags);
}
u32 kvm_get_inst(u32 *opc, struct kvm_vcpu *vcpu)
{
struct mips_coproc *cop0 = vcpu->arch.cop0;
unsigned long paddr, flags, vpn2, asid;
unsigned long va = (unsigned long)opc;
void *vaddr;
u32 inst;
int index;
if (KVM_GUEST_KSEGX(va) < KVM_GUEST_KSEG0 ||
KVM_GUEST_KSEGX(va) == KVM_GUEST_KSEG23) {
local_irq_save(flags);
index = kvm_mips_host_tlb_lookup(vcpu, va);
if (index >= 0) {
inst = *(opc);
} else {
vpn2 = va & VPN2_MASK;
asid = kvm_read_c0_guest_entryhi(cop0) &
KVM_ENTRYHI_ASID;
index = kvm_mips_guest_tlb_lookup(vcpu, vpn2 | asid);
if (index < 0) {
kvm_err("%s: get_user_failed for %p, vcpu: %p, ASID: %#lx\n",
__func__, opc, vcpu, read_c0_entryhi());
kvm_mips_dump_host_tlbs();
kvm_mips_dump_guest_tlbs(vcpu);
local_irq_restore(flags);
return KVM_INVALID_INST;
}
kvm_mips_handle_mapped_seg_tlb_fault(vcpu,
&vcpu->arch.
guest_tlb[index]);
inst = *(opc);
}
local_irq_restore(flags);
} else if (KVM_GUEST_KSEGX(va) == KVM_GUEST_KSEG0) {
paddr = kvm_mips_translate_guest_kseg0_to_hpa(vcpu, va);
vaddr = kmap_atomic(pfn_to_page(PHYS_PFN(paddr)));
vaddr += paddr & ~PAGE_MASK;
inst = *(u32 *)vaddr;
kunmap_atomic(vaddr);
} else {
kvm_err("%s: illegal address: %p\n", __func__, opc);
return KVM_INVALID_INST;
}
return inst;
}

21
arch/mips/kvm/stats.c
View File

@ -11,27 +11,6 @@
#include <linux/kvm_host.h>
char *kvm_mips_exit_types_str[MAX_KVM_MIPS_EXIT_TYPES] = {
"WAIT",
"CACHE",
"Signal",
"Interrupt",
"COP0/1 Unusable",
"TLB Mod",
"TLB Miss (LD)",
"TLB Miss (ST)",
"Address Err (ST)",
"Address Error (LD)",
"System Call",
"Reserved Inst",
"Break Inst",
"Trap Inst",
"MSA FPE",
"FPE",
"MSA Disabled",
"D-Cache Flushes",
};
char *kvm_cop0_str[N_MIPS_COPROC_REGS] = {
"Index",
"Random",

498
arch/mips/kvm/tlb.c
View File

@ -14,7 +14,7 @@
#include <linux/smp.h>
#include <linux/mm.h>
#include <linux/delay.h>
#include <linux/module.h>
#include <linux/export.h>
#include <linux/kvm_host.h>
#include <linux/srcu.h>
@ -24,6 +24,7 @@
#include <asm/pgtable.h>
#include <asm/cacheflush.h>
#include <asm/tlb.h>
#include <asm/tlbdebug.h>
#undef CONFIG_MIPS_MT
#include <asm/r4kcache.h>
@ -32,22 +33,10 @@
#define KVM_GUEST_PC_TLB 0
#define KVM_GUEST_SP_TLB 1
#define PRIx64 "llx"
atomic_t kvm_mips_instance;
EXPORT_SYMBOL_GPL(kvm_mips_instance);
/* These function pointers are initialized once the KVM module is loaded */
kvm_pfn_t (*kvm_mips_gfn_to_pfn)(struct kvm *kvm, gfn_t gfn);
EXPORT_SYMBOL_GPL(kvm_mips_gfn_to_pfn);
void (*kvm_mips_release_pfn_clean)(kvm_pfn_t pfn);
EXPORT_SYMBOL_GPL(kvm_mips_release_pfn_clean);
bool (*kvm_mips_is_error_pfn)(kvm_pfn_t pfn);
EXPORT_SYMBOL_GPL(kvm_mips_is_error_pfn);
uint32_t kvm_mips_get_kernel_asid(struct kvm_vcpu *vcpu)
static u32 kvm_mips_get_kernel_asid(struct kvm_vcpu *vcpu)
{
int cpu = smp_processor_id();
@ -55,7 +44,7 @@ uint32_t kvm_mips_get_kernel_asid(struct kvm_vcpu *vcpu)
cpu_asid_mask(&cpu_data[cpu]);
}
uint32_t kvm_mips_get_user_asid(struct kvm_vcpu *vcpu)
static u32 kvm_mips_get_user_asid(struct kvm_vcpu *vcpu)
{
int cpu = smp_processor_id();
@ -63,7 +52,7 @@ uint32_t kvm_mips_get_user_asid(struct kvm_vcpu *vcpu)
cpu_asid_mask(&cpu_data[cpu]);
}
inline uint32_t kvm_mips_get_commpage_asid(struct kvm_vcpu *vcpu)
inline u32 kvm_mips_get_commpage_asid(struct kvm_vcpu *vcpu)
{
return vcpu->kvm->arch.commpage_tlb;
}
@ -72,50 +61,15 @@ inline uint32_t kvm_mips_get_commpage_asid(struct kvm_vcpu *vcpu)
void kvm_mips_dump_host_tlbs(void)
{
unsigned long old_entryhi;
unsigned long old_pagemask;
struct kvm_mips_tlb tlb;
unsigned long flags;
int i;
local_irq_save(flags);
old_entryhi = read_c0_entryhi();
old_pagemask = read_c0_pagemask();
kvm_info("HOST TLBs:\n");
kvm_info("ASID: %#lx\n", read_c0_entryhi() &
cpu_asid_mask(&current_cpu_data));
dump_tlb_regs();
pr_info("\n");
dump_tlb_all();
for (i = 0; i < current_cpu_data.tlbsize; i++) {
write_c0_index(i);
mtc0_tlbw_hazard();
tlb_read();
tlbw_use_hazard();
tlb.tlb_hi = read_c0_entryhi();
tlb.tlb_lo0 = read_c0_entrylo0();
tlb.tlb_lo1 = read_c0_entrylo1();
tlb.tlb_mask = read_c0_pagemask();
kvm_info("TLB%c%3d Hi 0x%08lx ",
(tlb.tlb_lo0 | tlb.tlb_lo1) & MIPS3_PG_V ? ' ' : '*',
i, tlb.tlb_hi);
kvm_info("Lo0=0x%09" PRIx64 " %c%c attr %lx ",
(uint64_t) mips3_tlbpfn_to_paddr(tlb.tlb_lo0),
(tlb.tlb_lo0 & MIPS3_PG_D) ? 'D' : ' ',
(tlb.tlb_lo0 & MIPS3_PG_G) ? 'G' : ' ',
(tlb.tlb_lo0 >> 3) & 7);
kvm_info("Lo1=0x%09" PRIx64 " %c%c attr %lx sz=%lx\n",
(uint64_t) mips3_tlbpfn_to_paddr(tlb.tlb_lo1),
(tlb.tlb_lo1 & MIPS3_PG_D) ? 'D' : ' ',
(tlb.tlb_lo1 & MIPS3_PG_G) ? 'G' : ' ',
(tlb.tlb_lo1 >> 3) & 7, tlb.tlb_mask);
}
write_c0_entryhi(old_entryhi);
write_c0_pagemask(old_pagemask);
mtc0_tlbw_hazard();
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(kvm_mips_dump_host_tlbs);
@ -132,74 +86,24 @@ void kvm_mips_dump_guest_tlbs(struct kvm_vcpu *vcpu)
for (i = 0; i < KVM_MIPS_GUEST_TLB_SIZE; i++) {
tlb = vcpu->arch.guest_tlb[i];
kvm_info("TLB%c%3d Hi 0x%08lx ",
(tlb.tlb_lo0 | tlb.tlb_lo1) & MIPS3_PG_V ? ' ' : '*',
(tlb.tlb_lo[0] | tlb.tlb_lo[1]) & ENTRYLO_V
? ' ' : '*',
i, tlb.tlb_hi);
kvm_info("Lo0=0x%09" PRIx64 " %c%c attr %lx ",
(uint64_t) mips3_tlbpfn_to_paddr(tlb.tlb_lo0),
(tlb.tlb_lo0 & MIPS3_PG_D) ? 'D' : ' ',
(tlb.tlb_lo0 & MIPS3_PG_G) ? 'G' : ' ',
(tlb.tlb_lo0 >> 3) & 7);
kvm_info("Lo1=0x%09" PRIx64 " %c%c attr %lx sz=%lx\n",
(uint64_t) mips3_tlbpfn_to_paddr(tlb.tlb_lo1),
(tlb.tlb_lo1 & MIPS3_PG_D) ? 'D' : ' ',
(tlb.tlb_lo1 & MIPS3_PG_G) ? 'G' : ' ',
(tlb.tlb_lo1 >> 3) & 7, tlb.tlb_mask);
kvm_info("Lo0=0x%09llx %c%c attr %lx ",
(u64) mips3_tlbpfn_to_paddr(tlb.tlb_lo[0]),
(tlb.tlb_lo[0] & ENTRYLO_D) ? 'D' : ' ',
(tlb.tlb_lo[0] & ENTRYLO_G) ? 'G' : ' ',
(tlb.tlb_lo[0] & ENTRYLO_C) >> ENTRYLO_C_SHIFT);
kvm_info("Lo1=0x%09llx %c%c attr %lx sz=%lx\n",
(u64) mips3_tlbpfn_to_paddr(tlb.tlb_lo[1]),
(tlb.tlb_lo[1] & ENTRYLO_D) ? 'D' : ' ',
(tlb.tlb_lo[1] & ENTRYLO_G) ? 'G' : ' ',
(tlb.tlb_lo[1] & ENTRYLO_C) >> ENTRYLO_C_SHIFT,
tlb.tlb_mask);
}
}
EXPORT_SYMBOL_GPL(kvm_mips_dump_guest_tlbs);
static int kvm_mips_map_page(struct kvm *kvm, gfn_t gfn)
{
int srcu_idx, err = 0;
kvm_pfn_t pfn;
if (kvm->arch.guest_pmap[gfn] != KVM_INVALID_PAGE)
return 0;
srcu_idx = srcu_read_lock(&kvm->srcu);
pfn = kvm_mips_gfn_to_pfn(kvm, gfn);
if (kvm_mips_is_error_pfn(pfn)) {
kvm_err("Couldn't get pfn for gfn %#" PRIx64 "!\n", gfn);
err = -EFAULT;
goto out;
}
kvm->arch.guest_pmap[gfn] = pfn;
out:
srcu_read_unlock(&kvm->srcu, srcu_idx);
return err;
}
/* Translate guest KSEG0 addresses to Host PA */
unsigned long kvm_mips_translate_guest_kseg0_to_hpa(struct kvm_vcpu *vcpu,
unsigned long gva)
{
gfn_t gfn;
uint32_t offset = gva & ~PAGE_MASK;
struct kvm *kvm = vcpu->kvm;
if (KVM_GUEST_KSEGX(gva) != KVM_GUEST_KSEG0) {
kvm_err("%s/%p: Invalid gva: %#lx\n", __func__,
__builtin_return_address(0), gva);
return KVM_INVALID_PAGE;
}
gfn = (KVM_GUEST_CPHYSADDR(gva) >> PAGE_SHIFT);
if (gfn >= kvm->arch.guest_pmap_npages) {
kvm_err("%s: Invalid gfn: %#llx, GVA: %#lx\n", __func__, gfn,
gva);
return KVM_INVALID_PAGE;
}
if (kvm_mips_map_page(vcpu->kvm, gfn) < 0)
return KVM_INVALID_ADDR;
return (kvm->arch.guest_pmap[gfn] << PAGE_SHIFT) + offset;
}
EXPORT_SYMBOL_GPL(kvm_mips_translate_guest_kseg0_to_hpa);
/* XXXKYMA: Must be called with interrupts disabled */
/* set flush_dcache_mask == 0 if no dcache flush required */
int kvm_mips_host_tlb_write(struct kvm_vcpu *vcpu, unsigned long entryhi,
@ -243,12 +147,12 @@ int kvm_mips_host_tlb_write(struct kvm_vcpu *vcpu, unsigned long entryhi,
/* Flush D-cache */
if (flush_dcache_mask) {
if (entrylo0 & MIPS3_PG_V) {
if (entrylo0 & ENTRYLO_V) {
++vcpu->stat.flush_dcache_exits;
flush_data_cache_page((entryhi & VPN2_MASK) &
~flush_dcache_mask);
}
if (entrylo1 & MIPS3_PG_V) {
if (entrylo1 & ENTRYLO_V) {
++vcpu->stat.flush_dcache_exits;
flush_data_cache_page(((entryhi & VPN2_MASK) &
~flush_dcache_mask) |
@ -259,96 +163,35 @@ int kvm_mips_host_tlb_write(struct kvm_vcpu *vcpu, unsigned long entryhi,
/* Restore old ASID */
write_c0_entryhi(old_entryhi);
mtc0_tlbw_hazard();
tlbw_use_hazard();
local_irq_restore(flags);
return 0;
}
/* XXXKYMA: Must be called with interrupts disabled */
int kvm_mips_handle_kseg0_tlb_fault(unsigned long badvaddr,
struct kvm_vcpu *vcpu)
{
gfn_t gfn;
kvm_pfn_t pfn0, pfn1;
unsigned long vaddr = 0;
unsigned long entryhi = 0, entrylo0 = 0, entrylo1 = 0;
int even;
struct kvm *kvm = vcpu->kvm;
const int flush_dcache_mask = 0;
int ret;
if (KVM_GUEST_KSEGX(badvaddr) != KVM_GUEST_KSEG0) {
kvm_err("%s: Invalid BadVaddr: %#lx\n", __func__, badvaddr);
kvm_mips_dump_host_tlbs();
return -1;
}
gfn = (KVM_GUEST_CPHYSADDR(badvaddr) >> PAGE_SHIFT);
if (gfn >= kvm->arch.guest_pmap_npages) {
kvm_err("%s: Invalid gfn: %#llx, BadVaddr: %#lx\n", __func__,
gfn, badvaddr);
kvm_mips_dump_host_tlbs();
return -1;
}
even = !(gfn & 0x1);
vaddr = badvaddr & (PAGE_MASK << 1);
if (kvm_mips_map_page(vcpu->kvm, gfn) < 0)
return -1;
if (kvm_mips_map_page(vcpu->kvm, gfn ^ 0x1) < 0)
return -1;
if (even) {
pfn0 = kvm->arch.guest_pmap[gfn];
pfn1 = kvm->arch.guest_pmap[gfn ^ 0x1];
} else {
pfn0 = kvm->arch.guest_pmap[gfn ^ 0x1];
pfn1 = kvm->arch.guest_pmap[gfn];
}
entrylo0 = mips3_paddr_to_tlbpfn(pfn0 << PAGE_SHIFT) | (0x3 << 3) |
(1 << 2) | (0x1 << 1);
entrylo1 = mips3_paddr_to_tlbpfn(pfn1 << PAGE_SHIFT) | (0x3 << 3) |
(1 << 2) | (0x1 << 1);
preempt_disable();
entryhi = (vaddr | kvm_mips_get_kernel_asid(vcpu));
ret = kvm_mips_host_tlb_write(vcpu, entryhi, entrylo0, entrylo1,
flush_dcache_mask);
preempt_enable();
return ret;
}
EXPORT_SYMBOL_GPL(kvm_mips_handle_kseg0_tlb_fault);
EXPORT_SYMBOL_GPL(kvm_mips_host_tlb_write);
int kvm_mips_handle_commpage_tlb_fault(unsigned long badvaddr,
struct kvm_vcpu *vcpu)
{
kvm_pfn_t pfn0, pfn1;
kvm_pfn_t pfn;
unsigned long flags, old_entryhi = 0, vaddr = 0;
unsigned long entrylo0 = 0, entrylo1 = 0;
unsigned long entrylo[2] = { 0, 0 };
unsigned int pair_idx;
pfn0 = CPHYSADDR(vcpu->arch.kseg0_commpage) >> PAGE_SHIFT;
pfn1 = 0;
entrylo0 = mips3_paddr_to_tlbpfn(pfn0 << PAGE_SHIFT) | (0x3 << 3) |
(1 << 2) | (0x1 << 1);
entrylo1 = 0;
pfn = PFN_DOWN(virt_to_phys(vcpu->arch.kseg0_commpage));
pair_idx = (badvaddr >> PAGE_SHIFT) & 1;
entrylo[pair_idx] = mips3_paddr_to_tlbpfn(pfn << PAGE_SHIFT) |
((_page_cachable_default >> _CACHE_SHIFT) << ENTRYLO_C_SHIFT) |
ENTRYLO_D | ENTRYLO_V;
local_irq_save(flags);
old_entryhi = read_c0_entryhi();
vaddr = badvaddr & (PAGE_MASK << 1);
write_c0_entryhi(vaddr | kvm_mips_get_kernel_asid(vcpu));
mtc0_tlbw_hazard();
write_c0_entrylo0(entrylo0);
mtc0_tlbw_hazard();
write_c0_entrylo1(entrylo1);
mtc0_tlbw_hazard();
write_c0_entrylo0(entrylo[0]);
write_c0_entrylo1(entrylo[1]);
write_c0_index(kvm_mips_get_commpage_asid(vcpu));
mtc0_tlbw_hazard();
tlb_write_indexed();
mtc0_tlbw_hazard();
tlbw_use_hazard();
kvm_debug("@ %#lx idx: %2d [entryhi(R): %#lx] entrylo0 (R): 0x%08lx, entrylo1(R): 0x%08lx\n",
@ -358,68 +201,12 @@ int kvm_mips_handle_commpage_tlb_fault(unsigned long badvaddr,
/* Restore old ASID */
write_c0_entryhi(old_entryhi);
mtc0_tlbw_hazard();
tlbw_use_hazard();
local_irq_restore(flags);
return 0;
}
EXPORT_SYMBOL_GPL(kvm_mips_handle_commpage_tlb_fault);
int kvm_mips_handle_mapped_seg_tlb_fault(struct kvm_vcpu *vcpu,
struct kvm_mips_tlb *tlb,
unsigned long *hpa0,
unsigned long *hpa1)
{
unsigned long entryhi = 0, entrylo0 = 0, entrylo1 = 0;
struct kvm *kvm = vcpu->kvm;
kvm_pfn_t pfn0, pfn1;
int ret;
if ((tlb->tlb_hi & VPN2_MASK) == 0) {
pfn0 = 0;
pfn1 = 0;
} else {
if (kvm_mips_map_page(kvm, mips3_tlbpfn_to_paddr(tlb->tlb_lo0)
>> PAGE_SHIFT) < 0)
return -1;
if (kvm_mips_map_page(kvm, mips3_tlbpfn_to_paddr(tlb->tlb_lo1)
>> PAGE_SHIFT) < 0)
return -1;
pfn0 = kvm->arch.guest_pmap[mips3_tlbpfn_to_paddr(tlb->tlb_lo0)
>> PAGE_SHIFT];
pfn1 = kvm->arch.guest_pmap[mips3_tlbpfn_to_paddr(tlb->tlb_lo1)
>> PAGE_SHIFT];
}
if (hpa0)
*hpa0 = pfn0 << PAGE_SHIFT;
if (hpa1)
*hpa1 = pfn1 << PAGE_SHIFT;
/* Get attributes from the Guest TLB */
entrylo0 = mips3_paddr_to_tlbpfn(pfn0 << PAGE_SHIFT) | (0x3 << 3) |
(tlb->tlb_lo0 & MIPS3_PG_D) | (tlb->tlb_lo0 & MIPS3_PG_V);
entrylo1 = mips3_paddr_to_tlbpfn(pfn1 << PAGE_SHIFT) | (0x3 << 3) |
(tlb->tlb_lo1 & MIPS3_PG_D) | (tlb->tlb_lo1 & MIPS3_PG_V);
kvm_debug("@ %#lx tlb_lo0: 0x%08lx tlb_lo1: 0x%08lx\n", vcpu->arch.pc,
tlb->tlb_lo0, tlb->tlb_lo1);
preempt_disable();
entryhi = (tlb->tlb_hi & VPN2_MASK) | (KVM_GUEST_KERNEL_MODE(vcpu) ?
kvm_mips_get_kernel_asid(vcpu) :
kvm_mips_get_user_asid(vcpu));
ret = kvm_mips_host_tlb_write(vcpu, entryhi, entrylo0, entrylo1,
tlb->tlb_mask);
preempt_enable();
return ret;
}
EXPORT_SYMBOL_GPL(kvm_mips_handle_mapped_seg_tlb_fault);
int kvm_mips_guest_tlb_lookup(struct kvm_vcpu *vcpu, unsigned long entryhi)
{
int i;
@ -435,7 +222,7 @@ int kvm_mips_guest_tlb_lookup(struct kvm_vcpu *vcpu, unsigned long entryhi)
}
kvm_debug("%s: entryhi: %#lx, index: %d lo0: %#lx, lo1: %#lx\n",
__func__, entryhi, index, tlb[i].tlb_lo0, tlb[i].tlb_lo1);
__func__, entryhi, index, tlb[i].tlb_lo[0], tlb[i].tlb_lo[1]);
return index;
}
@ -467,7 +254,6 @@ int kvm_mips_host_tlb_lookup(struct kvm_vcpu *vcpu, unsigned long vaddr)
/* Restore old ASID */
write_c0_entryhi(old_entryhi);
mtc0_tlbw_hazard();
tlbw_use_hazard();
local_irq_restore(flags);
@ -498,21 +284,16 @@ int kvm_mips_host_tlb_inv(struct kvm_vcpu *vcpu, unsigned long va)
if (idx > 0) {
write_c0_entryhi(UNIQUE_ENTRYHI(idx));
mtc0_tlbw_hazard();
write_c0_entrylo0(0);
mtc0_tlbw_hazard();
write_c0_entrylo1(0);
mtc0_tlbw_hazard();
tlb_write_indexed();
mtc0_tlbw_hazard();
tlbw_use_hazard();
}
write_c0_entryhi(old_entryhi);
mtc0_tlbw_hazard();
tlbw_use_hazard();
local_irq_restore(flags);
@ -540,61 +321,39 @@ void kvm_mips_flush_host_tlb(int skip_kseg0)
/* Blast 'em all away. */
for (entry = 0; entry < maxentry; entry++) {
write_c0_index(entry);
mtc0_tlbw_hazard();
if (skip_kseg0) {
mtc0_tlbr_hazard();
tlb_read();
tlbw_use_hazard();
tlb_read_hazard();
entryhi = read_c0_entryhi();
/* Don't blow away guest kernel entries */
if (KVM_GUEST_KSEGX(entryhi) == KVM_GUEST_KSEG0)
continue;
write_c0_pagemask(old_pagemask);
}
/* Make sure all entries differ. */
write_c0_entryhi(UNIQUE_ENTRYHI(entry));
mtc0_tlbw_hazard();
write_c0_entrylo0(0);
mtc0_tlbw_hazard();
write_c0_entrylo1(0);
mtc0_tlbw_hazard();
tlb_write_indexed();
mtc0_tlbw_hazard();
tlbw_use_hazard();
}
tlbw_use_hazard();
write_c0_entryhi(old_entryhi);
write_c0_pagemask(old_pagemask);
mtc0_tlbw_hazard();
tlbw_use_hazard();
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(kvm_mips_flush_host_tlb);
void kvm_get_new_mmu_context(struct mm_struct *mm, unsigned long cpu,
struct kvm_vcpu *vcpu)
{
unsigned long asid = asid_cache(cpu);
asid += cpu_asid_inc();
if (!(asid & cpu_asid_mask(&cpu_data[cpu]))) {
if (cpu_has_vtag_icache)
flush_icache_all();
kvm_local_flush_tlb_all(); /* start new asid cycle */
if (!asid) /* fix version if needed */
asid = asid_first_version(cpu);
}
cpu_context(cpu, mm) = asid_cache(cpu) = asid;
}
void kvm_local_flush_tlb_all(void)
{
unsigned long flags;
@ -614,185 +373,12 @@ void kvm_local_flush_tlb_all(void)
write_c0_index(entry);
mtc0_tlbw_hazard();
tlb_write_indexed();
tlbw_use_hazard();
entry++;
}
tlbw_use_hazard();
write_c0_entryhi(old_ctx);
mtc0_tlbw_hazard();
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(kvm_local_flush_tlb_all);
/**
* kvm_mips_migrate_count() - Migrate timer.
* @vcpu: Virtual CPU.
*
* Migrate CP0_Count hrtimer to the current CPU by cancelling and restarting it
* if it was running prior to being cancelled.
*
* Must be called when the VCPU is migrated to a different CPU to ensure that
* timer expiry during guest execution interrupts the guest and causes the
* interrupt to be delivered in a timely manner.
*/
static void kvm_mips_migrate_count(struct kvm_vcpu *vcpu)
{
if (hrtimer_cancel(&vcpu->arch.comparecount_timer))
hrtimer_restart(&vcpu->arch.comparecount_timer);
}
/* Restore ASID once we are scheduled back after preemption */
void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
{
unsigned long asid_mask = cpu_asid_mask(&cpu_data[cpu]);
unsigned long flags;
int newasid = 0;
kvm_debug("%s: vcpu %p, cpu: %d\n", __func__, vcpu, cpu);
/* Allocate new kernel and user ASIDs if needed */
local_irq_save(flags);
if ((vcpu->arch.guest_kernel_asid[cpu] ^ asid_cache(cpu)) &
asid_version_mask(cpu)) {
kvm_get_new_mmu_context(&vcpu->arch.guest_kernel_mm, cpu, vcpu);
vcpu->arch.guest_kernel_asid[cpu] =
vcpu->arch.guest_kernel_mm.context.asid[cpu];
kvm_get_new_mmu_context(&vcpu->arch.guest_user_mm, cpu, vcpu);
vcpu->arch.guest_user_asid[cpu] =
vcpu->arch.guest_user_mm.context.asid[cpu];
newasid++;
kvm_debug("[%d]: cpu_context: %#lx\n", cpu,
cpu_context(cpu, current->mm));
kvm_debug("[%d]: Allocated new ASID for Guest Kernel: %#x\n",
cpu, vcpu->arch.guest_kernel_asid[cpu]);
kvm_debug("[%d]: Allocated new ASID for Guest User: %#x\n", cpu,
vcpu->arch.guest_user_asid[cpu]);
}
if (vcpu->arch.last_sched_cpu != cpu) {
kvm_debug("[%d->%d]KVM VCPU[%d] switch\n",
vcpu->arch.last_sched_cpu, cpu, vcpu->vcpu_id);
/*
* Migrate the timer interrupt to the current CPU so that it
* always interrupts the guest and synchronously triggers a
* guest timer interrupt.
*/
kvm_mips_migrate_count(vcpu);
}
if (!newasid) {
/*
* If we preempted while the guest was executing, then reload
* the pre-empted ASID
*/
if (current->flags & PF_VCPU) {
write_c0_entryhi(vcpu->arch.
preempt_entryhi & asid_mask);
ehb();
}
} else {
/* New ASIDs were allocated for the VM */
/*
* Were we in guest context? If so then the pre-empted ASID is
* no longer valid, we need to set it to what it should be based
* on the mode of the Guest (Kernel/User)
*/
if (current->flags & PF_VCPU) {
if (KVM_GUEST_KERNEL_MODE(vcpu))
write_c0_entryhi(vcpu->arch.
guest_kernel_asid[cpu] &
asid_mask);
else
write_c0_entryhi(vcpu->arch.
guest_user_asid[cpu] &
asid_mask);
ehb();
}
}
/* restore guest state to registers */
kvm_mips_callbacks->vcpu_set_regs(vcpu);
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(kvm_arch_vcpu_load);
/* ASID can change if another task is scheduled during preemption */
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
{
unsigned long flags;
uint32_t cpu;
local_irq_save(flags);
cpu = smp_processor_id();
vcpu->arch.preempt_entryhi = read_c0_entryhi();
vcpu->arch.last_sched_cpu = cpu;
/* save guest state in registers */
kvm_mips_callbacks->vcpu_get_regs(vcpu);
if (((cpu_context(cpu, current->mm) ^ asid_cache(cpu)) &
asid_version_mask(cpu))) {
kvm_debug("%s: Dropping MMU Context: %#lx\n", __func__,
cpu_context(cpu, current->mm));
drop_mmu_context(current->mm, cpu);
}
write_c0_entryhi(cpu_asid(cpu, current->mm));
ehb();
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(kvm_arch_vcpu_put);
uint32_t kvm_get_inst(uint32_t *opc, struct kvm_vcpu *vcpu)
{
struct mips_coproc *cop0 = vcpu->arch.cop0;
unsigned long paddr, flags, vpn2, asid;
uint32_t inst;
int index;
if (KVM_GUEST_KSEGX((unsigned long) opc) < KVM_GUEST_KSEG0 ||
KVM_GUEST_KSEGX((unsigned long) opc) == KVM_GUEST_KSEG23) {
local_irq_save(flags);
index = kvm_mips_host_tlb_lookup(vcpu, (unsigned long) opc);
if (index >= 0) {
inst = *(opc);
} else {
vpn2 = (unsigned long) opc & VPN2_MASK;
asid = kvm_read_c0_guest_entryhi(cop0) &
KVM_ENTRYHI_ASID;
index = kvm_mips_guest_tlb_lookup(vcpu, vpn2 | asid);
if (index < 0) {
kvm_err("%s: get_user_failed for %p, vcpu: %p, ASID: %#lx\n",
__func__, opc, vcpu, read_c0_entryhi());
kvm_mips_dump_host_tlbs();
local_irq_restore(flags);
return KVM_INVALID_INST;
}
kvm_mips_handle_mapped_seg_tlb_fault(vcpu,
&vcpu->arch.
guest_tlb[index],
NULL, NULL);
inst = *(opc);
}
local_irq_restore(flags);
} else if (KVM_GUEST_KSEGX(opc) == KVM_GUEST_KSEG0) {
paddr =
kvm_mips_translate_guest_kseg0_to_hpa(vcpu,
(unsigned long) opc);
inst = *(uint32_t *) CKSEG0ADDR(paddr);
} else {
kvm_err("%s: illegal address: %p\n", __func__, opc);
return KVM_INVALID_INST;
}
return inst;
}
EXPORT_SYMBOL_GPL(kvm_get_inst);

236
arch/mips/kvm/trace.h
View File

@ -17,8 +17,75 @@
#define TRACE_INCLUDE_PATH .
#define TRACE_INCLUDE_FILE trace
/* Tracepoints for VM eists */
extern char *kvm_mips_exit_types_str[MAX_KVM_MIPS_EXIT_TYPES];
/*
* Tracepoints for VM enters
*/
DECLARE_EVENT_CLASS(kvm_transition,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu),
TP_STRUCT__entry(
__field(unsigned long, pc)
),
TP_fast_assign(
__entry->pc = vcpu->arch.pc;
),
TP_printk("PC: 0x%08lx",
__entry->pc)
);
DEFINE_EVENT(kvm_transition, kvm_enter,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu));
DEFINE_EVENT(kvm_transition, kvm_reenter,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu));
DEFINE_EVENT(kvm_transition, kvm_out,
TP_PROTO(struct kvm_vcpu *vcpu),
TP_ARGS(vcpu));
/* The first 32 exit reasons correspond to Cause.ExcCode */
#define KVM_TRACE_EXIT_INT 0
#define KVM_TRACE_EXIT_TLBMOD 1
#define KVM_TRACE_EXIT_TLBMISS_LD 2
#define KVM_TRACE_EXIT_TLBMISS_ST 3
#define KVM_TRACE_EXIT_ADDRERR_LD 4
#define KVM_TRACE_EXIT_ADDRERR_ST 5
#define KVM_TRACE_EXIT_SYSCALL 8
#define KVM_TRACE_EXIT_BREAK_INST 9
#define KVM_TRACE_EXIT_RESVD_INST 10
#define KVM_TRACE_EXIT_COP_UNUSABLE 11
#define KVM_TRACE_EXIT_TRAP_INST 13
#define KVM_TRACE_EXIT_MSA_FPE 14
#define KVM_TRACE_EXIT_FPE 15
#define KVM_TRACE_EXIT_MSA_DISABLED 21
/* Further exit reasons */
#define KVM_TRACE_EXIT_WAIT 32
#define KVM_TRACE_EXIT_CACHE 33
#define KVM_TRACE_EXIT_SIGNAL 34
/* Tracepoints for VM exits */
#define kvm_trace_symbol_exit_types \
{ KVM_TRACE_EXIT_INT, "Interrupt" }, \
{ KVM_TRACE_EXIT_TLBMOD, "TLB Mod" }, \
{ KVM_TRACE_EXIT_TLBMISS_LD, "TLB Miss (LD)" }, \
{ KVM_TRACE_EXIT_TLBMISS_ST, "TLB Miss (ST)" }, \
{ KVM_TRACE_EXIT_ADDRERR_LD, "Address Error (LD)" }, \
{ KVM_TRACE_EXIT_ADDRERR_ST, "Address Err (ST)" }, \
{ KVM_TRACE_EXIT_SYSCALL, "System Call" }, \
{ KVM_TRACE_EXIT_BREAK_INST, "Break Inst" }, \
{ KVM_TRACE_EXIT_RESVD_INST, "Reserved Inst" }, \
{ KVM_TRACE_EXIT_COP_UNUSABLE, "COP0/1 Unusable" }, \
{ KVM_TRACE_EXIT_TRAP_INST, "Trap Inst" }, \
{ KVM_TRACE_EXIT_MSA_FPE, "MSA FPE" }, \
{ KVM_TRACE_EXIT_FPE, "FPE" }, \
{ KVM_TRACE_EXIT_MSA_DISABLED, "MSA Disabled" }, \
{ KVM_TRACE_EXIT_WAIT, "WAIT" }, \
{ KVM_TRACE_EXIT_CACHE, "CACHE" }, \
{ KVM_TRACE_EXIT_SIGNAL, "Signal" }
TRACE_EVENT(kvm_exit,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int reason),
@ -34,10 +101,173 @@ TRACE_EVENT(kvm_exit,
),
TP_printk("[%s]PC: 0x%08lx",
kvm_mips_exit_types_str[__entry->reason],
__print_symbolic(__entry->reason,
kvm_trace_symbol_exit_types),
__entry->pc)
);
#define KVM_TRACE_MFC0 0
#define KVM_TRACE_MTC0 1
#define KVM_TRACE_DMFC0 2
#define KVM_TRACE_DMTC0 3
#define KVM_TRACE_RDHWR 4
#define KVM_TRACE_HWR_COP0 0
#define KVM_TRACE_HWR_HWR 1
#define KVM_TRACE_COP0(REG, SEL) ((KVM_TRACE_HWR_COP0 << 8) | \
((REG) << 3) | (SEL))
#define KVM_TRACE_HWR(REG, SEL) ((KVM_TRACE_HWR_HWR << 8) | \
((REG) << 3) | (SEL))
#define kvm_trace_symbol_hwr_ops \
{ KVM_TRACE_MFC0, "MFC0" }, \
{ KVM_TRACE_MTC0, "MTC0" }, \
{ KVM_TRACE_DMFC0, "DMFC0" }, \
{ KVM_TRACE_DMTC0, "DMTC0" }, \
{ KVM_TRACE_RDHWR, "RDHWR" }
#define kvm_trace_symbol_hwr_cop \
{ KVM_TRACE_HWR_COP0, "COP0" }, \
{ KVM_TRACE_HWR_HWR, "HWR" }
#define kvm_trace_symbol_hwr_regs \
{ KVM_TRACE_COP0( 0, 0), "Index" }, \
{ KVM_TRACE_COP0( 2, 0), "EntryLo0" }, \
{ KVM_TRACE_COP0( 3, 0), "EntryLo1" }, \
{ KVM_TRACE_COP0( 4, 0), "Context" }, \
{ KVM_TRACE_COP0( 4, 2), "UserLocal" }, \
{ KVM_TRACE_COP0( 5, 0), "PageMask" }, \
{ KVM_TRACE_COP0( 6, 0), "Wired" }, \
{ KVM_TRACE_COP0( 7, 0), "HWREna" }, \
{ KVM_TRACE_COP0( 8, 0), "BadVAddr" }, \
{ KVM_TRACE_COP0( 9, 0), "Count" }, \
{ KVM_TRACE_COP0(10, 0), "EntryHi" }, \
{ KVM_TRACE_COP0(11, 0), "Compare" }, \
{ KVM_TRACE_COP0(12, 0), "Status" }, \
{ KVM_TRACE_COP0(12, 1), "IntCtl" }, \
{ KVM_TRACE_COP0(12, 2), "SRSCtl" }, \
{ KVM_TRACE_COP0(13, 0), "Cause" }, \
{ KVM_TRACE_COP0(14, 0), "EPC" }, \
{ KVM_TRACE_COP0(15, 0), "PRId" }, \
{ KVM_TRACE_COP0(15, 1), "EBase" }, \
{ KVM_TRACE_COP0(16, 0), "Config" }, \
{ KVM_TRACE_COP0(16, 1), "Config1" }, \
{ KVM_TRACE_COP0(16, 2), "Config2" }, \
{ KVM_TRACE_COP0(16, 3), "Config3" }, \
{ KVM_TRACE_COP0(16, 4), "Config4" }, \
{ KVM_TRACE_COP0(16, 5), "Config5" }, \
{ KVM_TRACE_COP0(16, 7), "Config7" }, \
{ KVM_TRACE_COP0(26, 0), "ECC" }, \
{ KVM_TRACE_COP0(30, 0), "ErrorEPC" }, \
{ KVM_TRACE_COP0(31, 2), "KScratch1" }, \
{ KVM_TRACE_COP0(31, 3), "KScratch2" }, \
{ KVM_TRACE_COP0(31, 4), "KScratch3" }, \
{ KVM_TRACE_COP0(31, 5), "KScratch4" }, \
{ KVM_TRACE_COP0(31, 6), "KScratch5" }, \
{ KVM_TRACE_COP0(31, 7), "KScratch6" }, \
{ KVM_TRACE_HWR( 0, 0), "CPUNum" }, \
{ KVM_TRACE_HWR( 1, 0), "SYNCI_Step" }, \
{ KVM_TRACE_HWR( 2, 0), "CC" }, \
{ KVM_TRACE_HWR( 3, 0), "CCRes" }, \
{ KVM_TRACE_HWR(29, 0), "ULR" }
TRACE_EVENT(kvm_hwr,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int op, unsigned int reg,
unsigned long val),
TP_ARGS(vcpu, op, reg, val),
TP_STRUCT__entry(
__field(unsigned long, val)
__field(u16, reg)
__field(u8, op)
),
TP_fast_assign(
__entry->val = val;
__entry->reg = reg;
__entry->op = op;
),
TP_printk("%s %s (%s:%u:%u) 0x%08lx",
__print_symbolic(__entry->op,
kvm_trace_symbol_hwr_ops),
__print_symbolic(__entry->reg,
kvm_trace_symbol_hwr_regs),
__print_symbolic(__entry->reg >> 8,
kvm_trace_symbol_hwr_cop),
(__entry->reg >> 3) & 0x1f,
__entry->reg & 0x7,
__entry->val)
);
#define KVM_TRACE_AUX_RESTORE 0
#define KVM_TRACE_AUX_SAVE 1
#define KVM_TRACE_AUX_ENABLE 2
#define KVM_TRACE_AUX_DISABLE 3
#define KVM_TRACE_AUX_DISCARD 4
#define KVM_TRACE_AUX_FPU 1
#define KVM_TRACE_AUX_MSA 2
#define KVM_TRACE_AUX_FPU_MSA 3
#define kvm_trace_symbol_aux_op \
{ KVM_TRACE_AUX_RESTORE, "restore" }, \
{ KVM_TRACE_AUX_SAVE, "save" }, \
{ KVM_TRACE_AUX_ENABLE, "enable" }, \
{ KVM_TRACE_AUX_DISABLE, "disable" }, \
{ KVM_TRACE_AUX_DISCARD, "discard" }
#define kvm_trace_symbol_aux_state \
{ KVM_TRACE_AUX_FPU, "FPU" }, \
{ KVM_TRACE_AUX_MSA, "MSA" }, \
{ KVM_TRACE_AUX_FPU_MSA, "FPU & MSA" }
TRACE_EVENT(kvm_aux,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int op,
unsigned int state),
TP_ARGS(vcpu, op, state),
TP_STRUCT__entry(
__field(unsigned long, pc)
__field(u8, op)
__field(u8, state)
),
TP_fast_assign(
__entry->pc = vcpu->arch.pc;
__entry->op = op;
__entry->state = state;
),
TP_printk("%s %s PC: 0x%08lx",
__print_symbolic(__entry->op,
kvm_trace_symbol_aux_op),
__print_symbolic(__entry->state,
kvm_trace_symbol_aux_state),
__entry->pc)
);
TRACE_EVENT(kvm_asid_change,
TP_PROTO(struct kvm_vcpu *vcpu, unsigned int old_asid,
unsigned int new_asid),
TP_ARGS(vcpu, old_asid, new_asid),
TP_STRUCT__entry(
__field(unsigned long, pc)
__field(u8, old_asid)
__field(u8, new_asid)
),
TP_fast_assign(
__entry->pc = vcpu->arch.pc;
__entry->old_asid = old_asid;
__entry->new_asid = new_asid;
),
TP_printk("PC: 0x%08lx old: 0x%02x new: 0x%02x",
__entry->pc,
__entry->old_asid,
__entry->new_asid)
);
#endif /* _TRACE_KVM_H */
/* This part must be outside protection */

178
arch/mips/kvm/trap_emul.c
View File

@ -21,7 +21,7 @@
static gpa_t kvm_trap_emul_gva_to_gpa_cb(gva_t gva)
{
gpa_t gpa;
uint32_t kseg = KSEGX(gva);
gva_t kseg = KSEGX(gva);
if ((kseg == CKSEG0) || (kseg == CKSEG1))
gpa = CPHYSADDR(gva);
@ -40,8 +40,8 @@ static int kvm_trap_emul_handle_cop_unusable(struct kvm_vcpu *vcpu)
{
struct mips_coproc *cop0 = vcpu->arch.cop0;
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -87,15 +87,15 @@ static int kvm_trap_emul_handle_cop_unusable(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_tlb_mod(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
unsigned long badvaddr = vcpu->arch.host_cp0_badvaddr;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
if (KVM_GUEST_KSEGX(badvaddr) < KVM_GUEST_KSEG0
|| KVM_GUEST_KSEGX(badvaddr) == KVM_GUEST_KSEG23) {
kvm_debug("USER/KSEG23 ADDR TLB MOD fault: cause %#lx, PC: %p, BadVaddr: %#lx\n",
kvm_debug("USER/KSEG23 ADDR TLB MOD fault: cause %#x, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
er = kvm_mips_handle_tlbmod(cause, opc, run, vcpu);
@ -111,14 +111,14 @@ static int kvm_trap_emul_handle_tlb_mod(struct kvm_vcpu *vcpu)
* when we are not using HIGHMEM. Need to address this in a
* HIGHMEM kernel
*/
kvm_err("TLB MOD fault not handled, cause %#lx, PC: %p, BadVaddr: %#lx\n",
kvm_err("TLB MOD fault not handled, cause %#x, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
kvm_mips_dump_host_tlbs();
kvm_arch_vcpu_dump_regs(vcpu);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
} else {
kvm_err("Illegal TLB Mod fault address , cause %#lx, PC: %p, BadVaddr: %#lx\n",
kvm_err("Illegal TLB Mod fault address , cause %#x, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
kvm_mips_dump_host_tlbs();
kvm_arch_vcpu_dump_regs(vcpu);
@ -128,12 +128,12 @@ static int kvm_trap_emul_handle_tlb_mod(struct kvm_vcpu *vcpu)
return ret;
}
static int kvm_trap_emul_handle_tlb_st_miss(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_tlb_miss(struct kvm_vcpu *vcpu, bool store)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
unsigned long badvaddr = vcpu->arch.host_cp0_badvaddr;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -145,55 +145,8 @@ static int kvm_trap_emul_handle_tlb_st_miss(struct kvm_vcpu *vcpu)
}
} else if (KVM_GUEST_KSEGX(badvaddr) < KVM_GUEST_KSEG0
|| KVM_GUEST_KSEGX(badvaddr) == KVM_GUEST_KSEG23) {
kvm_debug("USER ADDR TLB LD fault: cause %#lx, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
er = kvm_mips_handle_tlbmiss(cause, opc, run, vcpu);
if (er == EMULATE_DONE)
ret = RESUME_GUEST;
else {
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
}
} else if (KVM_GUEST_KSEGX(badvaddr) == KVM_GUEST_KSEG0) {
/*
* All KSEG0 faults are handled by KVM, as the guest kernel does
* not expect to ever get them
*/
if (kvm_mips_handle_kseg0_tlb_fault
(vcpu->arch.host_cp0_badvaddr, vcpu) < 0) {
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
}
} else {
kvm_err("Illegal TLB LD fault address , cause %#lx, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
kvm_mips_dump_host_tlbs();
kvm_arch_vcpu_dump_regs(vcpu);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
}
return ret;
}
static int kvm_trap_emul_handle_tlb_ld_miss(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
unsigned long badvaddr = vcpu->arch.host_cp0_badvaddr;
unsigned long cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
if (((badvaddr & PAGE_MASK) == KVM_GUEST_COMMPAGE_ADDR)
&& KVM_GUEST_KERNEL_MODE(vcpu)) {
if (kvm_mips_handle_commpage_tlb_fault(badvaddr, vcpu) < 0) {
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
}
} else if (KVM_GUEST_KSEGX(badvaddr) < KVM_GUEST_KSEG0
|| KVM_GUEST_KSEGX(badvaddr) == KVM_GUEST_KSEG23) {
kvm_debug("USER ADDR TLB ST fault: PC: %#lx, BadVaddr: %#lx\n",
vcpu->arch.pc, badvaddr);
kvm_debug("USER ADDR TLB %s fault: cause %#x, PC: %p, BadVaddr: %#lx\n",
store ? "ST" : "LD", cause, opc, badvaddr);
/*
* User Address (UA) fault, this could happen if
@ -213,14 +166,18 @@ static int kvm_trap_emul_handle_tlb_ld_miss(struct kvm_vcpu *vcpu)
ret = RESUME_HOST;
}
} else if (KVM_GUEST_KSEGX(badvaddr) == KVM_GUEST_KSEG0) {
/*
* All KSEG0 faults are handled by KVM, as the guest kernel does
* not expect to ever get them
*/
if (kvm_mips_handle_kseg0_tlb_fault
(vcpu->arch.host_cp0_badvaddr, vcpu) < 0) {
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
}
} else {
kvm_err("Illegal TLB ST fault address , cause %#lx, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
kvm_err("Illegal TLB %s fault address , cause %#x, PC: %p, BadVaddr: %#lx\n",
store ? "ST" : "LD", cause, opc, badvaddr);
kvm_mips_dump_host_tlbs();
kvm_arch_vcpu_dump_regs(vcpu);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
@ -229,12 +186,22 @@ static int kvm_trap_emul_handle_tlb_ld_miss(struct kvm_vcpu *vcpu)
return ret;
}
static int kvm_trap_emul_handle_tlb_st_miss(struct kvm_vcpu *vcpu)
{
return kvm_trap_emul_handle_tlb_miss(vcpu, true);
}
static int kvm_trap_emul_handle_tlb_ld_miss(struct kvm_vcpu *vcpu)
{
return kvm_trap_emul_handle_tlb_miss(vcpu, false);
}
static int kvm_trap_emul_handle_addr_err_st(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
unsigned long badvaddr = vcpu->arch.host_cp0_badvaddr;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -251,7 +218,7 @@ static int kvm_trap_emul_handle_addr_err_st(struct kvm_vcpu *vcpu)
ret = RESUME_HOST;
}
} else {
kvm_err("Address Error (STORE): cause %#lx, PC: %p, BadVaddr: %#lx\n",
kvm_err("Address Error (STORE): cause %#x, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
@ -262,9 +229,9 @@ static int kvm_trap_emul_handle_addr_err_st(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_addr_err_ld(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
unsigned long badvaddr = vcpu->arch.host_cp0_badvaddr;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -280,7 +247,7 @@ static int kvm_trap_emul_handle_addr_err_ld(struct kvm_vcpu *vcpu)
ret = RESUME_HOST;
}
} else {
kvm_err("Address Error (LOAD): cause %#lx, PC: %p, BadVaddr: %#lx\n",
kvm_err("Address Error (LOAD): cause %#x, PC: %p, BadVaddr: %#lx\n",
cause, opc, badvaddr);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
ret = RESUME_HOST;
@ -292,8 +259,8 @@ static int kvm_trap_emul_handle_addr_err_ld(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_syscall(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -310,8 +277,8 @@ static int kvm_trap_emul_handle_syscall(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_res_inst(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -328,8 +295,8 @@ static int kvm_trap_emul_handle_res_inst(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_break(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -346,8 +313,8 @@ static int kvm_trap_emul_handle_break(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_trap(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *)vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *)vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -364,8 +331,8 @@ static int kvm_trap_emul_handle_trap(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_msa_fpe(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *)vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *)vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -382,8 +349,8 @@ static int kvm_trap_emul_handle_msa_fpe(struct kvm_vcpu *vcpu)
static int kvm_trap_emul_handle_fpe(struct kvm_vcpu *vcpu)
{
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *)vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *)vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -407,8 +374,8 @@ static int kvm_trap_emul_handle_msa_disabled(struct kvm_vcpu *vcpu)
{
struct mips_coproc *cop0 = vcpu->arch.cop0;
struct kvm_run *run = vcpu->run;
uint32_t __user *opc = (uint32_t __user *) vcpu->arch.pc;
unsigned long cause = vcpu->arch.host_cp0_cause;
u32 __user *opc = (u32 __user *) vcpu->arch.pc;
u32 cause = vcpu->arch.host_cp0_cause;
enum emulation_result er = EMULATE_DONE;
int ret = RESUME_GUEST;
@ -451,24 +418,41 @@ static int kvm_trap_emul_vm_init(struct kvm *kvm)
static int kvm_trap_emul_vcpu_init(struct kvm_vcpu *vcpu)
{
vcpu->arch.kscratch_enabled = 0xfc;
return 0;
}
static int kvm_trap_emul_vcpu_setup(struct kvm_vcpu *vcpu)
{
struct mips_coproc *cop0 = vcpu->arch.cop0;
uint32_t config1;
u32 config, config1;
int vcpu_id = vcpu->vcpu_id;
/*
* Arch specific stuff, set up config registers properly so that the
* guest will come up as expected, for now we simulate a MIPS 24kc
* guest will come up as expected
*/
#ifndef CONFIG_CPU_MIPSR6
/* r2-r5, simulate a MIPS 24kc */
kvm_write_c0_guest_prid(cop0, 0x00019300);
/* Have config1, Cacheable, noncoherent, write-back, write allocate */
kvm_write_c0_guest_config(cop0, MIPS_CONF_M | (0x3 << CP0C0_K0) |
(0x1 << CP0C0_AR) |
(MMU_TYPE_R4000 << CP0C0_MT));
#else
/* r6+, simulate a generic QEMU machine */
kvm_write_c0_guest_prid(cop0, 0x00010000);
#endif
/*
* Have config1, Cacheable, noncoherent, write-back, write allocate.
* Endianness, arch revision & virtually tagged icache should match
* host.
*/
config = read_c0_config() & MIPS_CONF_AR;
config |= MIPS_CONF_M | CONF_CM_CACHABLE_NONCOHERENT | MIPS_CONF_MT_TLB;
#ifdef CONFIG_CPU_BIG_ENDIAN
config |= CONF_BE;
#endif
if (cpu_has_vtag_icache)
config |= MIPS_CONF_VI;
kvm_write_c0_guest_config(cop0, config);
/* Read the cache characteristics from the host Config1 Register */
config1 = (read_c0_config1() & ~0x7f);
@ -478,9 +462,8 @@ static int kvm_trap_emul_vcpu_setup(struct kvm_vcpu *vcpu)
config1 |= ((KVM_MIPS_GUEST_TLB_SIZE - 1) << 25);
/* We unset some bits that we aren't emulating */
config1 &=
~((1 << CP0C1_C2) | (1 << CP0C1_MD) | (1 << CP0C1_PC) |
(1 << CP0C1_WR) | (1 << CP0C1_CA));
config1 &= ~(MIPS_CONF1_C2 | MIPS_CONF1_MD | MIPS_CONF1_PC |
MIPS_CONF1_WR | MIPS_CONF1_CA);
kvm_write_c0_guest_config1(cop0, config1);
/* Have config3, no tertiary/secondary caches implemented */
@ -511,6 +494,17 @@ static int kvm_trap_emul_vcpu_setup(struct kvm_vcpu *vcpu)
return 0;
}
static unsigned long kvm_trap_emul_num_regs(struct kvm_vcpu *vcpu)
{
return 0;
}
static int kvm_trap_emul_copy_reg_indices(struct kvm_vcpu *vcpu,
u64 __user *indices)
{
return 0;
}
static int kvm_trap_emul_get_one_reg(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg,
s64 *v)
@ -660,6 +654,8 @@ static struct kvm_mips_callbacks kvm_trap_emul_callbacks = {
.dequeue_io_int = kvm_mips_dequeue_io_int_cb,
.irq_deliver = kvm_mips_irq_deliver_cb,
.irq_clear = kvm_mips_irq_clear_cb,
.num_regs = kvm_trap_emul_num_regs,
.copy_reg_indices = kvm_trap_emul_copy_reg_indices,
.get_one_reg = kvm_trap_emul_get_one_reg,
.set_one_reg = kvm_trap_emul_set_one_reg,
.vcpu_get_regs = kvm_trap_emul_vcpu_get_regs,

8
arch/mips/math-emu/cp1emu.c
View File

@ -627,8 +627,8 @@ static int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn,
dec_insn.pc_inc +
dec_insn.next_pc_inc;
return 1;
case cbcond0_op:
case cbcond1_op:
case pop10_op:
case pop30_op:
if (!cpu_has_mips_r6)
break;
if (insn.i_format.rt && !insn.i_format.rs)
@ -683,14 +683,14 @@ static int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn,
dec_insn.next_pc_inc;
return 1;
case beqzcjic_op:
case pop66_op:
if (!cpu_has_mips_r6)
break;
*contpc = regs->cp0_epc + dec_insn.pc_inc +
dec_insn.next_pc_inc;
return 1;
case bnezcjialc_op:
case pop76_op:
if (!cpu_has_mips_r6)
break;
if (!insn.i_format.rs)

2
arch/mips/mm/c-r4k.c
View File

@ -1206,7 +1206,7 @@ static void probe_pcache(void)
c->icache.linesz;
c->icache.waybit = __ffs(icache_size/c->icache.ways);
if (config & 0x8) /* VI bit */
if (config & MIPS_CONF_VI)
c->icache.flags |= MIPS_CACHE_VTAG;
/*

13
arch/mips/mm/uasm-micromips.c
View File

@ -53,8 +53,13 @@ static struct insn insn_table_MM[] = {
{ insn_bltzl, 0, 0 },
{ insn_bne, M(mm_bne32_op, 0, 0, 0, 0, 0), RT | RS | BIMM },
{ insn_cache, M(mm_pool32b_op, 0, 0, mm_cache_func, 0, 0), RT | RS | SIMM },
{ insn_cfc1, M(mm_pool32f_op, 0, 0, 0, mm_cfc1_op, mm_32f_73_op), RT | RS },
{ insn_cfcmsa, M(mm_pool32s_op, 0, msa_cfc_op, 0, 0, mm_32s_elm_op), RD | RE },
{ insn_ctc1, M(mm_pool32f_op, 0, 0, 0, mm_ctc1_op, mm_32f_73_op), RT | RS },
{ insn_ctcmsa, M(mm_pool32s_op, 0, msa_ctc_op, 0, 0, mm_32s_elm_op), RD | RE },
{ insn_daddu, 0, 0 },
{ insn_daddiu, 0, 0 },
{ insn_di, M(mm_pool32a_op, 0, 0, 0, mm_di_op, mm_pool32axf_op), RS },
{ insn_divu, M(mm_pool32a_op, 0, 0, 0, mm_divu_op, mm_pool32axf_op), RT | RS },
{ insn_dmfc0, 0, 0 },
{ insn_dmtc0, 0, 0 },
@ -84,6 +89,8 @@ static struct insn insn_table_MM[] = {
{ insn_mfhi, M(mm_pool32a_op, 0, 0, 0, mm_mfhi32_op, mm_pool32axf_op), RS },
{ insn_mflo, M(mm_pool32a_op, 0, 0, 0, mm_mflo32_op, mm_pool32axf_op), RS },
{ insn_mtc0, M(mm_pool32a_op, 0, 0, 0, mm_mtc0_op, mm_pool32axf_op), RT | RS | RD },
{ insn_mthi, M(mm_pool32a_op, 0, 0, 0, mm_mthi32_op, mm_pool32axf_op), RS },
{ insn_mtlo, M(mm_pool32a_op, 0, 0, 0, mm_mtlo32_op, mm_pool32axf_op), RS },
{ insn_mul, M(mm_pool32a_op, 0, 0, 0, 0, mm_mul_op), RT | RS | RD },
{ insn_or, M(mm_pool32a_op, 0, 0, 0, 0, mm_or32_op), RT | RS | RD },
{ insn_ori, M(mm_ori32_op, 0, 0, 0, 0, 0), RT | RS | UIMM },
@ -166,13 +173,15 @@ static void build_insn(u32 **buf, enum opcode opc, ...)
op = ip->match;
va_start(ap, opc);
if (ip->fields & RS) {
if (opc == insn_mfc0 || opc == insn_mtc0)
if (opc == insn_mfc0 || opc == insn_mtc0 ||
opc == insn_cfc1 || opc == insn_ctc1)
op |= build_rt(va_arg(ap, u32));
else
op |= build_rs(va_arg(ap, u32));
}
if (ip->fields & RT) {
if (opc == insn_mfc0 || opc == insn_mtc0)
if (opc == insn_mfc0 || opc == insn_mtc0 ||
opc == insn_cfc1 || opc == insn_ctc1)
op |= build_rs(va_arg(ap, u32));
else
op |= build_rt(va_arg(ap, u32));

11
arch/mips/mm/uasm-mips.c
View File

@ -67,9 +67,14 @@ static struct insn insn_table[] = {
#else
{ insn_cache, M6(cache_op, 0, 0, 0, cache6_op), RS | RT | SIMM9 },
#endif
{ insn_cfc1, M(cop1_op, cfc_op, 0, 0, 0, 0), RT | RD },
{ insn_cfcmsa, M(msa_op, 0, msa_cfc_op, 0, 0, msa_elm_op), RD | RE },
{ insn_ctc1, M(cop1_op, ctc_op, 0, 0, 0, 0), RT | RD },
{ insn_ctcmsa, M(msa_op, 0, msa_ctc_op, 0, 0, msa_elm_op), RD | RE },
{ insn_daddiu, M(daddiu_op, 0, 0, 0, 0, 0), RS | RT | SIMM },
{ insn_daddu, M(spec_op, 0, 0, 0, 0, daddu_op), RS | RT | RD },
{ insn_dinsm, M(spec3_op, 0, 0, 0, 0, dinsm_op), RS | RT | RD | RE },
{ insn_di, M(cop0_op, mfmc0_op, 0, 12, 0, 0), RT },
{ insn_dins, M(spec3_op, 0, 0, 0, 0, dins_op), RS | RT | RD | RE },
{ insn_divu, M(spec_op, 0, 0, 0, 0, divu_op), RS | RT },
{ insn_dmfc0, M(cop0_op, dmfc_op, 0, 0, 0, 0), RT | RD | SET},
@ -114,7 +119,13 @@ static struct insn insn_table[] = {
{ insn_mflo, M(spec_op, 0, 0, 0, 0, mflo_op), RD },
{ insn_mtc0, M(cop0_op, mtc_op, 0, 0, 0, 0), RT | RD | SET},
{ insn_mthc0, M(cop0_op, mthc0_op, 0, 0, 0, 0), RT | RD | SET},
{ insn_mthi, M(spec_op, 0, 0, 0, 0, mthi_op), RS },
{ insn_mtlo, M(spec_op, 0, 0, 0, 0, mtlo_op), RS },
#ifndef CONFIG_CPU_MIPSR6
{ insn_mul, M(spec2_op, 0, 0, 0, 0, mul_op), RS | RT | RD},
#else
{ insn_mul, M(spec_op, 0, 0, 0, mult_mul_op, mult_op), RS | RT | RD},
#endif
{ insn_ori, M(ori_op, 0, 0, 0, 0, 0), RS | RT | UIMM },
{ insn_or, M(spec_op, 0, 0, 0, 0, or_op), RS | RT | RD },
#ifndef CONFIG_CPU_MIPSR6

24
arch/mips/mm/uasm.c
View File

@ -49,18 +49,19 @@ enum opcode {
insn_invalid,
insn_addiu, insn_addu, insn_and, insn_andi, insn_bbit0, insn_bbit1,
insn_beq, insn_beql, insn_bgez, insn_bgezl, insn_bltz, insn_bltzl,
insn_bne, insn_cache, insn_daddiu, insn_daddu, insn_dins, insn_dinsm,
insn_divu, insn_dmfc0, insn_dmtc0, insn_drotr, insn_drotr32, insn_dsll,
insn_bne, insn_cache, insn_cfc1, insn_cfcmsa, insn_ctc1, insn_ctcmsa,
insn_daddiu, insn_daddu, insn_di, insn_dins, insn_dinsm, insn_divu,
insn_dmfc0, insn_dmtc0, insn_drotr, insn_drotr32, insn_dsll,
insn_dsll32, insn_dsra, insn_dsrl, insn_dsrl32, insn_dsubu, insn_eret,
insn_ext, insn_ins, insn_j, insn_jal, insn_jalr, insn_jr, insn_lb,
insn_ld, insn_ldx, insn_lh, insn_ll, insn_lld, insn_lui, insn_lw,
insn_lwx, insn_mfc0, insn_mfhc0, insn_mfhi, insn_mflo, insn_mtc0,
insn_mthc0, insn_mul, insn_or, insn_ori, insn_pref, insn_rfe,
insn_rotr, insn_sc, insn_scd, insn_sd, insn_sll, insn_sllv, insn_slt,
insn_sltiu, insn_sltu, insn_sra, insn_srl, insn_srlv, insn_subu,
insn_sw, insn_sync, insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi,
insn_tlbwr, insn_wait, insn_wsbh, insn_xor, insn_xori, insn_yield,
insn_lddir, insn_ldpte,
insn_mthc0, insn_mthi, insn_mtlo, insn_mul, insn_or, insn_ori,
insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd, insn_sd, insn_sll,
insn_sllv, insn_slt, insn_sltiu, insn_sltu, insn_sra, insn_srl,
insn_srlv, insn_subu, insn_sw, insn_sync, insn_syscall, insn_tlbp,
insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait, insn_wsbh, insn_xor,
insn_xori, insn_yield, insn_lddir, insn_ldpte,
};
struct insn {
@ -268,10 +269,15 @@ I_u1s2(_bltz)
I_u1s2(_bltzl)
I_u1u2s3(_bne)
I_u2s3u1(_cache)
I_u1u2(_cfc1)
I_u2u1(_cfcmsa)
I_u1u2(_ctc1)
I_u2u1(_ctcmsa)
I_u1u2u3(_dmfc0)
I_u1u2u3(_dmtc0)
I_u2u1s3(_daddiu)
I_u3u1u2(_daddu)
I_u1(_di);
I_u1u2(_divu)
I_u2u1u3(_dsll)
I_u2u1u3(_dsll32)
@ -301,6 +307,8 @@ I_u1(_mfhi)
I_u1(_mflo)
I_u1u2u3(_mtc0)
I_u1u2u3(_mthc0)
I_u1(_mthi)
I_u1(_mtlo)
I_u3u1u2(_mul)
I_u2u1u3(_ori)
I_u3u1u2(_or)

45
arch/powerpc/include/asm/hmi.h Normal file
View File

@ -0,0 +1,45 @@
/*
* Hypervisor Maintenance Interrupt header file.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program.
*
* Copyright 2015 IBM Corporation
* Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
*/
#ifndef __ASM_PPC64_HMI_H__
#define __ASM_PPC64_HMI_H__
#ifdef CONFIG_PPC_BOOK3S_64
#define CORE_TB_RESYNC_REQ_BIT 63
#define MAX_SUBCORE_PER_CORE 4
/*
* sibling_subcore_state structure is used to co-ordinate all threads
* during HMI to avoid TB corruption. This structure is allocated once
* per each core and shared by all threads on that core.
*/
struct sibling_subcore_state {
unsigned long flags;
u8 in_guest[MAX_SUBCORE_PER_CORE];
};
extern void wait_for_subcore_guest_exit(void);
extern void wait_for_tb_resync(void);
#else
static inline void wait_for_subcore_guest_exit(void) { }
static inline void wait_for_tb_resync(void) { }
#endif
#endif /* __ASM_PPC64_HMI_H__ */

6
arch/powerpc/include/asm/paca.h
View File

@ -26,6 +26,7 @@
#include <asm/kvm_book3s_asm.h>
#endif
#include <asm/accounting.h>
#include <asm/hmi.h>
register struct paca_struct *local_paca asm("r13");
@ -182,6 +183,11 @@ struct paca_struct {
*/
u16 in_mce;
u8 hmi_event_available; /* HMI event is available */
/*
* Bitmap for sibling subcore status. See kvm/book3s_hv_ras.c for
* more details
*/
struct sibling_subcore_state *sibling_subcore_state;
#endif
/* Stuff for accurate time accounting */

2
arch/powerpc/kernel/Makefile
View File

@ -41,7 +41,7 @@ obj-$(CONFIG_VDSO32) += vdso32/
obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
obj-$(CONFIG_PPC_BOOK3S_64) += cpu_setup_ppc970.o cpu_setup_pa6t.o
obj-$(CONFIG_PPC_BOOK3S_64) += cpu_setup_power.o
obj-$(CONFIG_PPC_BOOK3S_64) += mce.o mce_power.o
obj-$(CONFIG_PPC_BOOK3S_64) += mce.o mce_power.o hmi.o
obj-$(CONFIG_PPC_BOOK3E_64) += exceptions-64e.o idle_book3e.o
obj-$(CONFIG_PPC64) += vdso64/
obj-$(CONFIG_ALTIVEC) += vecemu.o

4
arch/powerpc/kernel/exceptions-64s.S
View File

@ -671,6 +671,8 @@ BEGIN_FTR_SECTION
beq h_doorbell_common
cmpwi r3,0xea0
beq h_virt_irq_common
cmpwi r3,0xe60
beq hmi_exception_common
FTR_SECTION_ELSE
cmpwi r3,0xa00
beq doorbell_super_common
@ -1172,7 +1174,7 @@ fwnmi_data_area:
.globl hmi_exception_early
hmi_exception_early:
EXCEPTION_PROLOG_1(PACA_EXGEN, NOTEST, 0xe60)
EXCEPTION_PROLOG_1(PACA_EXGEN, KVMTEST, 0xe62)
mr r10,r1 /* Save r1 */
ld r1,PACAEMERGSP(r13) /* Use emergency stack */
subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */

56
arch/powerpc/kernel/hmi.c Normal file
View File

@ -0,0 +1,56 @@
/*
* Hypervisor Maintenance Interrupt (HMI) handling.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program.
*
* Copyright 2015 IBM Corporation
* Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
*/
#undef DEBUG
#include <linux/types.h>
#include <linux/compiler.h>
#include <asm/paca.h>
#include <asm/hmi.h>
void wait_for_subcore_guest_exit(void)
{
int i;
/*
* NULL bitmap pointer indicates that KVM module hasn't
* been loaded yet and hence no guests are running.
* If no KVM is in use, no need to co-ordinate among threads
* as all of them will always be in host and no one is going
* to modify TB other than the opal hmi handler.
* Hence, just return from here.
*/
if (!local_paca->sibling_subcore_state)
return;
for (i = 0; i < MAX_SUBCORE_PER_CORE; i++)
while (local_paca->sibling_subcore_state->in_guest[i])
cpu_relax();
}
void wait_for_tb_resync(void)
{
if (!local_paca->sibling_subcore_state)
return;
while (test_bit(CORE_TB_RESYNC_REQ_BIT,
&local_paca->sibling_subcore_state->flags))
cpu_relax();
}

4
arch/powerpc/kernel/idle_book3s.S
View File

@ -336,7 +336,9 @@ ALT_FTR_SECTION_END_NESTED_IFSET(CPU_FTR_ARCH_207S, 66); \
ld r2,PACATOC(r13); \
ld r1,PACAR1(r13); \
std r3,ORIG_GPR3(r1); /* Save original r3 */ \
bl opal_rm_handle_hmi; \
li r3,0; /* NULL argument */ \
bl hmi_exception_realmode; \
nop; \
ld r3,ORIG_GPR3(r1); /* Restore original r3 */ \
20: nop;

5
arch/powerpc/kernel/traps.c
View File

@ -61,6 +61,7 @@
#include <asm/tm.h>
#include <asm/debug.h>
#include <asm/asm-prototypes.h>
#include <asm/hmi.h>
#include <sysdev/fsl_pci.h>
#if defined(CONFIG_DEBUGGER) || defined(CONFIG_KEXEC)
@ -308,9 +309,13 @@ long hmi_exception_realmode(struct pt_regs *regs)
{
__this_cpu_inc(irq_stat.hmi_exceptions);
wait_for_subcore_guest_exit();
if (ppc_md.hmi_exception_early)
ppc_md.hmi_exception_early(regs);
wait_for_tb_resync();
return 0;
}

41
arch/powerpc/kvm/book3s_hv.c
View File

@ -52,6 +52,7 @@
#include <asm/switch_to.h>
#include <asm/smp.h>
#include <asm/dbell.h>
#include <asm/hmi.h>
#include <linux/gfp.h>
#include <linux/vmalloc.h>
#include <linux/highmem.h>
@ -2522,7 +2523,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)
list_for_each_entry(pvc, &core_info.vcs[sub], preempt_list)
spin_unlock(&pvc->lock);
kvm_guest_enter();
guest_enter();
srcu_idx = srcu_read_lock(&vc->kvm->srcu);
@ -2570,7 +2571,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)
/* make sure updates to secondary vcpu structs are visible now */
smp_mb();
kvm_guest_exit();
guest_exit();
for (sub = 0; sub < core_info.n_subcores; ++sub)
list_for_each_entry_safe(pvc, vcnext, &core_info.vcs[sub],
@ -3401,6 +3402,38 @@ static struct kvmppc_ops kvm_ops_hv = {
.hcall_implemented = kvmppc_hcall_impl_hv,
};
static int kvm_init_subcore_bitmap(void)
{
int i, j;
int nr_cores = cpu_nr_cores();
struct sibling_subcore_state *sibling_subcore_state;
for (i = 0; i < nr_cores; i++) {
int first_cpu = i * threads_per_core;
int node = cpu_to_node(first_cpu);
/* Ignore if it is already allocated. */
if (paca[first_cpu].sibling_subcore_state)
continue;
sibling_subcore_state =
kmalloc_node(sizeof(struct sibling_subcore_state),
GFP_KERNEL, node);
if (!sibling_subcore_state)
return -ENOMEM;
memset(sibling_subcore_state, 0,
sizeof(struct sibling_subcore_state));
for (j = 0; j < threads_per_core; j++) {
int cpu = first_cpu + j;
paca[cpu].sibling_subcore_state = sibling_subcore_state;
}
}
return 0;
}
static int kvmppc_book3s_init_hv(void)
{
int r;
@ -3411,6 +3444,10 @@ static int kvmppc_book3s_init_hv(void)
if (r < 0)
return -ENODEV;
r = kvm_init_subcore_bitmap();
if (r)
return r;
kvm_ops_hv.owner = THIS_MODULE;
kvmppc_hv_ops = &kvm_ops_hv;

176
arch/powerpc/kvm/book3s_hv_ras.c
View File

@ -13,6 +13,9 @@
#include <linux/kernel.h>
#include <asm/opal.h>
#include <asm/mce.h>
#include <asm/machdep.h>
#include <asm/cputhreads.h>
#include <asm/hmi.h>
/* SRR1 bits for machine check on POWER7 */
#define SRR1_MC_LDSTERR (1ul << (63-42))
@ -140,3 +143,176 @@ long kvmppc_realmode_machine_check(struct kvm_vcpu *vcpu)
{
return kvmppc_realmode_mc_power7(vcpu);
}
/* Check if dynamic split is in force and return subcore size accordingly. */
static inline int kvmppc_cur_subcore_size(void)
{
if (local_paca->kvm_hstate.kvm_split_mode)
return local_paca->kvm_hstate.kvm_split_mode->subcore_size;
return threads_per_subcore;
}
void kvmppc_subcore_enter_guest(void)
{
int thread_id, subcore_id;
thread_id = cpu_thread_in_core(local_paca->paca_index);
subcore_id = thread_id / kvmppc_cur_subcore_size();
local_paca->sibling_subcore_state->in_guest[subcore_id] = 1;
}
void kvmppc_subcore_exit_guest(void)
{
int thread_id, subcore_id;
thread_id = cpu_thread_in_core(local_paca->paca_index);
subcore_id = thread_id / kvmppc_cur_subcore_size();
local_paca->sibling_subcore_state->in_guest[subcore_id] = 0;
}
static bool kvmppc_tb_resync_required(void)
{
if (test_and_set_bit(CORE_TB_RESYNC_REQ_BIT,
&local_paca->sibling_subcore_state->flags))
return false;
return true;
}
static void kvmppc_tb_resync_done(void)
{
clear_bit(CORE_TB_RESYNC_REQ_BIT,
&local_paca->sibling_subcore_state->flags);
}
/*
* kvmppc_realmode_hmi_handler() is called only by primary thread during
* guest exit path.
*
* There are multiple reasons why HMI could occur, one of them is
* Timebase (TB) error. If this HMI is due to TB error, then TB would
* have been in stopped state. The opal hmi handler Will fix it and
* restore the TB value with host timebase value. For HMI caused due
* to non-TB errors, opal hmi handler will not touch/restore TB register
* and hence there won't be any change in TB value.
*
* Since we are not sure about the cause of this HMI, we can't be sure
* about the content of TB register whether it holds guest or host timebase
* value. Hence the idea is to resync the TB on every HMI, so that we
* know about the exact state of the TB value. Resync TB call will
* restore TB to host timebase.
*
* Things to consider:
* - On TB error, HMI interrupt is reported on all the threads of the core
* that has encountered TB error irrespective of split-core mode.
* - The very first thread on the core that get chance to fix TB error
* would rsync the TB with local chipTOD value.
* - The resync TB is a core level action i.e. it will sync all the TBs
* in that core independent of split-core mode. This means if we trigger
* TB sync from a thread from one subcore, it would affect TB values of
* sibling subcores of the same core.
*
* All threads need to co-ordinate before making opal hmi handler.
* All threads will use sibling_subcore_state->in_guest[] (shared by all
* threads in the core) in paca which holds information about whether
* sibling subcores are in Guest mode or host mode. The in_guest[] array
* is of size MAX_SUBCORE_PER_CORE=4, indexed using subcore id to set/unset
* subcore status. Only primary threads from each subcore is responsible
* to set/unset its designated array element while entering/exiting the
* guset.
*
* After invoking opal hmi handler call, one of the thread (of entire core)
* will need to resync the TB. Bit 63 from subcore state bitmap flags
* (sibling_subcore_state->flags) will be used to co-ordinate between
* primary threads to decide who takes up the responsibility.
*
* This is what we do:
* - Primary thread from each subcore tries to set resync required bit[63]
* of paca->sibling_subcore_state->flags.
* - The first primary thread that is able to set the flag takes the
* responsibility of TB resync. (Let us call it as thread leader)
* - All other threads which are in host will call
* wait_for_subcore_guest_exit() and wait for in_guest[0-3] from
* paca->sibling_subcore_state to get cleared.
* - All the primary thread will clear its subcore status from subcore
* state in_guest[] array respectively.
* - Once all primary threads clear in_guest[0-3], all of them will invoke
* opal hmi handler.
* - Now all threads will wait for TB resync to complete by invoking
* wait_for_tb_resync() except the thread leader.
* - Thread leader will do a TB resync by invoking opal_resync_timebase()
* call and the it will clear the resync required bit.
* - All other threads will now come out of resync wait loop and proceed
* with individual execution.
* - On return of this function, primary thread will signal all
* secondary threads to proceed.
* - All secondary threads will eventually call opal hmi handler on
* their exit path.
*/
long kvmppc_realmode_hmi_handler(void)
{
int ptid = local_paca->kvm_hstate.ptid;
bool resync_req;
/* This is only called on primary thread. */
BUG_ON(ptid != 0);
__this_cpu_inc(irq_stat.hmi_exceptions);
/*
* By now primary thread has already completed guest->host
* partition switch but haven't signaled secondaries yet.
* All the secondary threads on this subcore is waiting
* for primary thread to signal them to go ahead.
*
* For threads from subcore which isn't in guest, they all will
* wait until all other subcores on this core exit the guest.
*
* Now set the resync required bit. If you are the first to
* set this bit then kvmppc_tb_resync_required() function will
* return true. For rest all other subcores
* kvmppc_tb_resync_required() will return false.
*
* If resync_req == true, then this thread is responsible to
* initiate TB resync after hmi handler has completed.
* All other threads on this core will wait until this thread
* clears the resync required bit flag.
*/
resync_req = kvmppc_tb_resync_required();
/* Reset the subcore status to indicate it has exited guest */
kvmppc_subcore_exit_guest();
/*
* Wait for other subcores on this core to exit the guest.
* All the primary threads and threads from subcore that are
* not in guest will wait here until all subcores are out
* of guest context.
*/
wait_for_subcore_guest_exit();
/*
* At this point we are sure that primary threads from each
* subcore on this core have completed guest->host partition
* switch. Now it is safe to call HMI handler.
*/
if (ppc_md.hmi_exception_early)
ppc_md.hmi_exception_early(NULL);
/*
* Check if this thread is responsible to resync TB.
* All other threads will wait until this thread completes the
* TB resync.
*/
if (resync_req) {
opal_resync_timebase();
/* Reset TB resync req bit */
kvmppc_tb_resync_done();
} else {
wait_for_tb_resync();
}
return 0;
}

527
arch/powerpc/kvm/book3s_hv_rmhandlers.S
View File

@ -29,6 +29,7 @@
#include <asm/kvm_book3s_asm.h>
#include <asm/book3s/64/mmu-hash.h>
#include <asm/tm.h>
#include <asm/opal.h>
#define VCPU_GPRS_TM(reg) (((reg) * ULONG_SIZE) + VCPU_GPR_TM)
@ -373,6 +374,18 @@ kvm_secondary_got_guest:
lwsync
std r0, HSTATE_KVM_VCORE(r13)
/*
* All secondaries exiting guest will fall through this path.
* Before proceeding, just check for HMI interrupt and
* invoke opal hmi handler. By now we are sure that the
* primary thread on this core/subcore has already made partition
* switch/TB resync and we are good to call opal hmi handler.
*/
cmpwi r12, BOOK3S_INTERRUPT_HMI
bne kvm_no_guest
li r3,0 /* NULL argument */
bl hmi_exception_realmode
/*
* At this point we have finished executing in the guest.
* We need to wait for hwthread_req to become zero, since
@ -427,6 +440,22 @@ kvm_no_guest:
* whole-core mode, so we need to nap.
*/
kvm_unsplit_nap:
/*
* When secondaries are napping in kvm_unsplit_nap() with
* hwthread_req = 1, HMI goes ignored even though subcores are
* already exited the guest. Hence HMI keeps waking up secondaries
* from nap in a loop and secondaries always go back to nap since
* no vcore is assigned to them. This makes impossible for primary
* thread to get hold of secondary threads resulting into a soft
* lockup in KVM path.
*
* Let us check if HMI is pending and handle it before we go to nap.
*/
cmpwi r12, BOOK3S_INTERRUPT_HMI
bne 55f
li r3, 0 /* NULL argument */
bl hmi_exception_realmode
55:
/*
* Ensure that secondary doesn't nap when it has
* its vcore pointer set.
@ -601,6 +630,11 @@ BEGIN_FTR_SECTION
mtspr SPRN_DPDES, r8
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
/* Mark the subcore state as inside guest */
bl kvmppc_subcore_enter_guest
nop
ld r5, HSTATE_KVM_VCORE(r13)
ld r4, HSTATE_KVM_VCPU(r13)
li r0,1
stb r0,VCORE_IN_GUEST(r5) /* signal secondaries to continue */
@ -655,112 +689,8 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
BEGIN_FTR_SECTION
b skip_tm
END_FTR_SECTION_IFCLR(CPU_FTR_TM)
/* Turn on TM/FP/VSX/VMX so we can restore them. */
mfmsr r5
li r6, MSR_TM >> 32
sldi r6, r6, 32
or r5, r5, r6
ori r5, r5, MSR_FP
oris r5, r5, (MSR_VEC | MSR_VSX)@h
mtmsrd r5
/*
* The user may change these outside of a transaction, so they must
* always be context switched.
*/
ld r5, VCPU_TFHAR(r4)
ld r6, VCPU_TFIAR(r4)
ld r7, VCPU_TEXASR(r4)
mtspr SPRN_TFHAR, r5
mtspr SPRN_TFIAR, r6
mtspr SPRN_TEXASR, r7
ld r5, VCPU_MSR(r4)
rldicl. r5, r5, 64 - MSR_TS_S_LG, 62
beq skip_tm /* TM not active in guest */
/* Make sure the failure summary is set, otherwise we'll program check
* when we trechkpt. It's possible that this might have been not set
* on a kvmppc_set_one_reg() call but we shouldn't let this crash the
* host.
*/
oris r7, r7, (TEXASR_FS)@h
mtspr SPRN_TEXASR, r7
/*
* We need to load up the checkpointed state for the guest.
* We need to do this early as it will blow away any GPRs, VSRs and
* some SPRs.
*/
mr r31, r4
addi r3, r31, VCPU_FPRS_TM
bl load_fp_state
addi r3, r31, VCPU_VRS_TM
bl load_vr_state
mr r4, r31
lwz r7, VCPU_VRSAVE_TM(r4)
mtspr SPRN_VRSAVE, r7
ld r5, VCPU_LR_TM(r4)
lwz r6, VCPU_CR_TM(r4)
ld r7, VCPU_CTR_TM(r4)
ld r8, VCPU_AMR_TM(r4)
ld r9, VCPU_TAR_TM(r4)
mtlr r5
mtcr r6
mtctr r7
mtspr SPRN_AMR, r8
mtspr SPRN_TAR, r9
/*
* Load up PPR and DSCR values but don't put them in the actual SPRs
* till the last moment to avoid running with userspace PPR and DSCR for
* too long.
*/
ld r29, VCPU_DSCR_TM(r4)
ld r30, VCPU_PPR_TM(r4)
std r2, PACATMSCRATCH(r13) /* Save TOC */
/* Clear the MSR RI since r1, r13 are all going to be foobar. */
li r5, 0
mtmsrd r5, 1
/* Load GPRs r0-r28 */
reg = 0
.rept 29
ld reg, VCPU_GPRS_TM(reg)(r31)
reg = reg + 1
.endr
mtspr SPRN_DSCR, r29
mtspr SPRN_PPR, r30
/* Load final GPRs */
ld 29, VCPU_GPRS_TM(29)(r31)
ld 30, VCPU_GPRS_TM(30)(r31)
ld 31, VCPU_GPRS_TM(31)(r31)
/* TM checkpointed state is now setup. All GPRs are now volatile. */
TRECHKPT
/* Now let's get back the state we need. */
HMT_MEDIUM
GET_PACA(r13)
ld r29, HSTATE_DSCR(r13)
mtspr SPRN_DSCR, r29
ld r4, HSTATE_KVM_VCPU(r13)
ld r1, HSTATE_HOST_R1(r13)
ld r2, PACATMSCRATCH(r13)
/* Set the MSR RI since we have our registers back. */
li r5, MSR_RI
mtmsrd r5, 1
skip_tm:
bl kvmppc_restore_tm
END_FTR_SECTION_IFSET(CPU_FTR_TM)
#endif
/* Load guest PMU registers */
@ -841,12 +771,6 @@ BEGIN_FTR_SECTION
/* Skip next section on POWER7 */
b 8f
END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
/* Turn on TM so we can access TFHAR/TFIAR/TEXASR */
mfmsr r8
li r0, 1
rldimi r8, r0, MSR_TM_LG, 63-MSR_TM_LG
mtmsrd r8
/* Load up POWER8-specific registers */
ld r5, VCPU_IAMR(r4)
lwz r6, VCPU_PSPB(r4)
@ -1436,106 +1360,8 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
BEGIN_FTR_SECTION
b 2f
END_FTR_SECTION_IFCLR(CPU_FTR_TM)
/* Turn on TM. */
mfmsr r8
li r0, 1
rldimi r8, r0, MSR_TM_LG, 63-MSR_TM_LG
mtmsrd r8
ld r5, VCPU_MSR(r9)
rldicl. r5, r5, 64 - MSR_TS_S_LG, 62
beq 1f /* TM not active in guest. */
li r3, TM_CAUSE_KVM_RESCHED
/* Clear the MSR RI since r1, r13 are all going to be foobar. */
li r5, 0
mtmsrd r5, 1
/* All GPRs are volatile at this point. */
TRECLAIM(R3)
/* Temporarily store r13 and r9 so we have some regs to play with */
SET_SCRATCH0(r13)
GET_PACA(r13)
std r9, PACATMSCRATCH(r13)
ld r9, HSTATE_KVM_VCPU(r13)
/* Get a few more GPRs free. */
std r29, VCPU_GPRS_TM(29)(r9)
std r30, VCPU_GPRS_TM(30)(r9)
std r31, VCPU_GPRS_TM(31)(r9)
/* Save away PPR and DSCR soon so don't run with user values. */
mfspr r31, SPRN_PPR
HMT_MEDIUM
mfspr r30, SPRN_DSCR
ld r29, HSTATE_DSCR(r13)
mtspr SPRN_DSCR, r29
/* Save all but r9, r13 & r29-r31 */
reg = 0
.rept 29
.if (reg != 9) && (reg != 13)
std reg, VCPU_GPRS_TM(reg)(r9)
.endif
reg = reg + 1
.endr
/* ... now save r13 */
GET_SCRATCH0(r4)
std r4, VCPU_GPRS_TM(13)(r9)
/* ... and save r9 */
ld r4, PACATMSCRATCH(r13)
std r4, VCPU_GPRS_TM(9)(r9)
/* Reload stack pointer and TOC. */
ld r1, HSTATE_HOST_R1(r13)
ld r2, PACATOC(r13)
/* Set MSR RI now we have r1 and r13 back. */
li r5, MSR_RI
mtmsrd r5, 1
/* Save away checkpinted SPRs. */
std r31, VCPU_PPR_TM(r9)
std r30, VCPU_DSCR_TM(r9)
mflr r5
mfcr r6
mfctr r7
mfspr r8, SPRN_AMR
mfspr r10, SPRN_TAR
std r5, VCPU_LR_TM(r9)
stw r6, VCPU_CR_TM(r9)
std r7, VCPU_CTR_TM(r9)
std r8, VCPU_AMR_TM(r9)
std r10, VCPU_TAR_TM(r9)
/* Restore r12 as trap number. */
lwz r12, VCPU_TRAP(r9)
/* Save FP/VSX. */
addi r3, r9, VCPU_FPRS_TM
bl store_fp_state
addi r3, r9, VCPU_VRS_TM
bl store_vr_state
mfspr r6, SPRN_VRSAVE
stw r6, VCPU_VRSAVE_TM(r9)
1:
/*
* We need to save these SPRs after the treclaim so that the software
* error code is recorded correctly in the TEXASR. Also the user may
* change these outside of a transaction, so they must always be
* context switched.
*/
mfspr r5, SPRN_TFHAR
mfspr r6, SPRN_TFIAR
mfspr r7, SPRN_TEXASR
std r5, VCPU_TFHAR(r9)
std r6, VCPU_TFIAR(r9)
std r7, VCPU_TEXASR(r9)
2:
bl kvmppc_save_tm
END_FTR_SECTION_IFSET(CPU_FTR_TM)
#endif
/* Increment yield count if they have a VPA */
@ -1683,6 +1509,23 @@ BEGIN_FTR_SECTION
mtspr SPRN_DPDES, r8
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
/* If HMI, call kvmppc_realmode_hmi_handler() */
cmpwi r12, BOOK3S_INTERRUPT_HMI
bne 27f
bl kvmppc_realmode_hmi_handler
nop
li r12, BOOK3S_INTERRUPT_HMI
/*
* At this point kvmppc_realmode_hmi_handler would have resync-ed
* the TB. Hence it is not required to subtract guest timebase
* offset from timebase. So, skip it.
*
* Also, do not call kvmppc_subcore_exit_guest() because it has
* been invoked as part of kvmppc_realmode_hmi_handler().
*/
b 30f
27:
/* Subtract timebase offset from timebase */
ld r8,VCORE_TB_OFFSET(r5)
cmpdi r8,0
@ -1698,8 +1541,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
addis r8,r8,0x100 /* if so, increment upper 40 bits */
mtspr SPRN_TBU40,r8
17: bl kvmppc_subcore_exit_guest
nop
30: ld r5,HSTATE_KVM_VCORE(r13)
ld r4,VCORE_KVM(r5) /* pointer to struct kvm */
/* Reset PCR */
17: ld r0, VCORE_PCR(r5)
ld r0, VCORE_PCR(r5)
cmpdi r0, 0
beq 18f
li r0, 0
@ -2245,6 +2093,13 @@ _GLOBAL(kvmppc_h_cede) /* r3 = vcpu pointer, r11 = msr, r13 = paca */
/* save FP state */
bl kvmppc_save_fp
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
BEGIN_FTR_SECTION
ld r9, HSTATE_KVM_VCPU(r13)
bl kvmppc_save_tm
END_FTR_SECTION_IFSET(CPU_FTR_TM)
#endif
/*
* Set DEC to the smaller of DEC and HDEC, so that we wake
* no later than the end of our timeslice (HDEC interrupts
@ -2321,6 +2176,12 @@ kvm_end_cede:
bl kvmhv_accumulate_time
#endif
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
BEGIN_FTR_SECTION
bl kvmppc_restore_tm
END_FTR_SECTION_IFSET(CPU_FTR_TM)
#endif
/* load up FP state */
bl kvmppc_load_fp
@ -2461,6 +2322,8 @@ BEGIN_FTR_SECTION
cmpwi r6, 3 /* hypervisor doorbell? */
beq 3f
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
cmpwi r6, 0xa /* Hypervisor maintenance ? */
beq 4f
li r3, 1 /* anything else, return 1 */
0: blr
@ -2482,6 +2345,11 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
li r3, -1
blr
/* Woken up due to Hypervisor maintenance interrupt */
4: li r12, BOOK3S_INTERRUPT_HMI
li r3, 1
blr
/*
* Determine what sort of external interrupt is pending (if any).
* Returns:
@ -2631,6 +2499,239 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
mr r4,r31
blr
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
/*
* Save transactional state and TM-related registers.
* Called with r9 pointing to the vcpu struct.
* This can modify all checkpointed registers, but
* restores r1, r2 and r9 (vcpu pointer) before exit.
*/
kvmppc_save_tm:
mflr r0
std r0, PPC_LR_STKOFF(r1)
/* Turn on TM. */
mfmsr r8
li r0, 1
rldimi r8, r0, MSR_TM_LG, 63-MSR_TM_LG
mtmsrd r8
ld r5, VCPU_MSR(r9)
rldicl. r5, r5, 64 - MSR_TS_S_LG, 62
beq 1f /* TM not active in guest. */
std r1, HSTATE_HOST_R1(r13)
li r3, TM_CAUSE_KVM_RESCHED
/* Clear the MSR RI since r1, r13 are all going to be foobar. */
li r5, 0
mtmsrd r5, 1
/* All GPRs are volatile at this point. */
TRECLAIM(R3)
/* Temporarily store r13 and r9 so we have some regs to play with */
SET_SCRATCH0(r13)
GET_PACA(r13)
std r9, PACATMSCRATCH(r13)
ld r9, HSTATE_KVM_VCPU(r13)
/* Get a few more GPRs free. */
std r29, VCPU_GPRS_TM(29)(r9)
std r30, VCPU_GPRS_TM(30)(r9)
std r31, VCPU_GPRS_TM(31)(r9)
/* Save away PPR and DSCR soon so don't run with user values. */
mfspr r31, SPRN_PPR
HMT_MEDIUM
mfspr r30, SPRN_DSCR
ld r29, HSTATE_DSCR(r13)
mtspr SPRN_DSCR, r29
/* Save all but r9, r13 & r29-r31 */
reg = 0
.rept 29
.if (reg != 9) && (reg != 13)
std reg, VCPU_GPRS_TM(reg)(r9)
.endif
reg = reg + 1
.endr
/* ... now save r13 */
GET_SCRATCH0(r4)
std r4, VCPU_GPRS_TM(13)(r9)
/* ... and save r9 */
ld r4, PACATMSCRATCH(r13)
std r4, VCPU_GPRS_TM(9)(r9)
/* Reload stack pointer and TOC. */
ld r1, HSTATE_HOST_R1(r13)
ld r2, PACATOC(r13)
/* Set MSR RI now we have r1 and r13 back. */
li r5, MSR_RI
mtmsrd r5, 1
/* Save away checkpinted SPRs. */
std r31, VCPU_PPR_TM(r9)
std r30, VCPU_DSCR_TM(r9)
mflr r5
mfcr r6
mfctr r7
mfspr r8, SPRN_AMR
mfspr r10, SPRN_TAR
std r5, VCPU_LR_TM(r9)
stw r6, VCPU_CR_TM(r9)
std r7, VCPU_CTR_TM(r9)
std r8, VCPU_AMR_TM(r9)
std r10, VCPU_TAR_TM(r9)
/* Restore r12 as trap number. */
lwz r12, VCPU_TRAP(r9)
/* Save FP/VSX. */
addi r3, r9, VCPU_FPRS_TM
bl store_fp_state
addi r3, r9, VCPU_VRS_TM
bl store_vr_state
mfspr r6, SPRN_VRSAVE
stw r6, VCPU_VRSAVE_TM(r9)
1:
/*
* We need to save these SPRs after the treclaim so that the software
* error code is recorded correctly in the TEXASR. Also the user may
* change these outside of a transaction, so they must always be
* context switched.
*/
mfspr r5, SPRN_TFHAR
mfspr r6, SPRN_TFIAR
mfspr r7, SPRN_TEXASR
std r5, VCPU_TFHAR(r9)
std r6, VCPU_TFIAR(r9)
std r7, VCPU_TEXASR(r9)
ld r0, PPC_LR_STKOFF(r1)
mtlr r0
blr
/*
* Restore transactional state and TM-related registers.
* Called with r4 pointing to the vcpu struct.
* This potentially modifies all checkpointed registers.
* It restores r1, r2, r4 from the PACA.
*/
kvmppc_restore_tm:
mflr r0
std r0, PPC_LR_STKOFF(r1)
/* Turn on TM/FP/VSX/VMX so we can restore them. */
mfmsr r5
li r6, MSR_TM >> 32
sldi r6, r6, 32
or r5, r5, r6
ori r5, r5, MSR_FP
oris r5, r5, (MSR_VEC | MSR_VSX)@h
mtmsrd r5
/*
* The user may change these outside of a transaction, so they must
* always be context switched.
*/
ld r5, VCPU_TFHAR(r4)
ld r6, VCPU_TFIAR(r4)
ld r7, VCPU_TEXASR(r4)
mtspr SPRN_TFHAR, r5
mtspr SPRN_TFIAR, r6
mtspr SPRN_TEXASR, r7
ld r5, VCPU_MSR(r4)
rldicl. r5, r5, 64 - MSR_TS_S_LG, 62
beqlr /* TM not active in guest */
std r1, HSTATE_HOST_R1(r13)
/* Make sure the failure summary is set, otherwise we'll program check
* when we trechkpt. It's possible that this might have been not set
* on a kvmppc_set_one_reg() call but we shouldn't let this crash the
* host.
*/
oris r7, r7, (TEXASR_FS)@h
mtspr SPRN_TEXASR, r7
/*
* We need to load up the checkpointed state for the guest.
* We need to do this early as it will blow away any GPRs, VSRs and
* some SPRs.
*/
mr r31, r4
addi r3, r31, VCPU_FPRS_TM
bl load_fp_state
addi r3, r31, VCPU_VRS_TM
bl load_vr_state
mr r4, r31
lwz r7, VCPU_VRSAVE_TM(r4)
mtspr SPRN_VRSAVE, r7
ld r5, VCPU_LR_TM(r4)
lwz r6, VCPU_CR_TM(r4)
ld r7, VCPU_CTR_TM(r4)
ld r8, VCPU_AMR_TM(r4)
ld r9, VCPU_TAR_TM(r4)
mtlr r5
mtcr r6
mtctr r7
mtspr SPRN_AMR, r8
mtspr SPRN_TAR, r9
/*
* Load up PPR and DSCR values but don't put them in the actual SPRs
* till the last moment to avoid running with userspace PPR and DSCR for
* too long.
*/
ld r29, VCPU_DSCR_TM(r4)
ld r30, VCPU_PPR_TM(r4)
std r2, PACATMSCRATCH(r13) /* Save TOC */
/* Clear the MSR RI since r1, r13 are all going to be foobar. */
li r5, 0
mtmsrd r5, 1
/* Load GPRs r0-r28 */
reg = 0
.rept 29
ld reg, VCPU_GPRS_TM(reg)(r31)
reg = reg + 1
.endr
mtspr SPRN_DSCR, r29
mtspr SPRN_PPR, r30
/* Load final GPRs */
ld 29, VCPU_GPRS_TM(29)(r31)
ld 30, VCPU_GPRS_TM(30)(r31)
ld 31, VCPU_GPRS_TM(31)(r31)
/* TM checkpointed state is now setup. All GPRs are now volatile. */
TRECHKPT
/* Now let's get back the state we need. */
HMT_MEDIUM
GET_PACA(r13)
ld r29, HSTATE_DSCR(r13)
mtspr SPRN_DSCR, r29
ld r4, HSTATE_KVM_VCPU(r13)
ld r1, HSTATE_HOST_R1(r13)
ld r2, PACATMSCRATCH(r13)
/* Set the MSR RI since we have our registers back. */
li r5, MSR_RI
mtmsrd r5, 1
ld r0, PPC_LR_STKOFF(r1)
mtlr r0
blr
#endif
/*
* We come here if we get any exception or interrupt while we are
* executing host real mode code while in guest MMU context.

16
arch/powerpc/kvm/book3s_pr.c
View File

@ -914,7 +914,7 @@ int kvmppc_handle_exit_pr(struct kvm_run *run, struct kvm_vcpu *vcpu,
/* We get here with MSR.EE=1 */
trace_kvm_exit(exit_nr, vcpu);
kvm_guest_exit();
guest_exit();
switch (exit_nr) {
case BOOK3S_INTERRUPT_INST_STORAGE:
@ -1049,7 +1049,17 @@ int kvmppc_handle_exit_pr(struct kvm_run *run, struct kvm_vcpu *vcpu,
int emul;
program_interrupt:
flags = vcpu->arch.shadow_srr1 & 0x1f0000ull;
/*
* shadow_srr1 only contains valid flags if we came here via
* a program exception. The other exceptions (emulation assist,
* FP unavailable, etc.) do not provide flags in SRR1, so use
* an illegal-instruction exception when injecting a program
* interrupt into the guest.
*/
if (exit_nr == BOOK3S_INTERRUPT_PROGRAM)
flags = vcpu->arch.shadow_srr1 & 0x1f0000ull;
else
flags = SRR1_PROGILL;
emul = kvmppc_get_last_inst(vcpu, INST_GENERIC, &last_inst);
if (emul != EMULATE_DONE) {
@ -1531,7 +1541,7 @@ static int kvmppc_vcpu_run_pr(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
kvmppc_clear_debug(vcpu);
/* No need for kvm_guest_exit. It's done in handle_exit.
/* No need for guest_exit. It's done in handle_exit.
We also get here with interrupts enabled. */
/* Make sure we save the guest FPU/Altivec/VSX state */

4
arch/powerpc/kvm/booke.c
View File

@ -776,7 +776,7 @@ int kvmppc_vcpu_run(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
ret = __kvmppc_vcpu_run(kvm_run, vcpu);
/* No need for kvm_guest_exit. It's done in handle_exit.
/* No need for guest_exit. It's done in handle_exit.
We also get here with interrupts enabled. */
/* Switch back to user space debug context */
@ -1012,7 +1012,7 @@ int kvmppc_handle_exit(struct kvm_run *run, struct kvm_vcpu *vcpu,
}
trace_kvm_exit(exit_nr, vcpu);
__kvm_guest_exit();
guest_exit_irqoff();
local_irq_enable();

1
arch/powerpc/kvm/emulate.c
View File

@ -302,7 +302,6 @@ int kvmppc_emulate_instruction(struct kvm_run *run, struct kvm_vcpu *vcpu)
advance = 0;
printk(KERN_ERR "Couldn't emulate instruction 0x%08x "
"(op %d xop %d)\n", inst, get_op(inst), get_xop(inst));
kvmppc_core_queue_program(vcpu, 0);
}
}

3
arch/powerpc/kvm/mpic.c
View File

@ -1823,7 +1823,8 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
return 0;
}
int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
int kvm_set_routing_entry(struct kvm *kvm,
struct kvm_kernel_irq_routing_entry *e,
const struct kvm_irq_routing_entry *ue)
{
int r = -EINVAL;

6
arch/powerpc/kvm/powerpc.c
View File

@ -119,7 +119,7 @@ int kvmppc_prepare_to_enter(struct kvm_vcpu *vcpu)
continue;
}
__kvm_guest_enter();
guest_enter_irqoff();
return 1;
}
@ -588,6 +588,10 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
r = 1;
break;
#endif
case KVM_CAP_PPC_HTM:
r = cpu_has_feature(CPU_FTR_TM_COMP) &&
is_kvmppc_hv_enabled(kvm);
break;
default:
r = 0;
break;

2
arch/powerpc/platforms/powernv/opal-wrappers.S
View File

@ -64,7 +64,6 @@ END_FTR_SECTION(0, 1); \
OPAL_BRANCH(opal_tracepoint_entry) \
mfcr r12; \
stw r12,8(r1); \
std r1,PACAR1(r13); \
li r11,0; \
mfmsr r12; \
ori r11,r11,MSR_EE; \
@ -127,7 +126,6 @@ opal_tracepoint_entry:
mfcr r12
std r11,16(r1)
stw r12,8(r1)
std r1,PACAR1(r13)
li r11,0
mfmsr r12
ori r11,r11,MSR_EE

377
arch/s390/hypfs/hypfs_diag.c
View File

@ -19,29 +19,10 @@
#include <asm/ebcdic.h>
#include "hypfs.h"
#define LPAR_NAME_LEN 8 /* lpar name len in diag 204 data */
#define CPU_NAME_LEN 16 /* type name len of cpus in diag224 name table */
#define TMP_SIZE 64 /* size of temporary buffers */
#define DBFS_D204_HDR_VERSION 0
/* diag 204 subcodes */
enum diag204_sc {
SUBC_STIB4 = 4,
SUBC_RSI = 5,
SUBC_STIB6 = 6,
SUBC_STIB7 = 7
};
/* The two available diag 204 data formats */
enum diag204_format {
INFO_SIMPLE = 0,
INFO_EXT = 0x00010000
};
/* bit is set in flags, when physical cpu info is included in diag 204 data */
#define LPAR_PHYS_FLG 0x80
static char *diag224_cpu_names; /* diag 224 name table */
static enum diag204_sc diag204_store_sc; /* used subcode for store */
static enum diag204_format diag204_info_type; /* used diag 204 data format */
@ -53,7 +34,7 @@ static int diag204_buf_pages; /* number of pages for diag204 data */
static struct dentry *dbfs_d204_file;
/*
* DIAG 204 data structures and member access functions.
* DIAG 204 member access functions.
*
* Since we have two different diag 204 data formats for old and new s390
* machines, we do not access the structs directly, but use getter functions for
@ -62,304 +43,173 @@ static struct dentry *dbfs_d204_file;
/* Time information block */
struct info_blk_hdr {
__u8 npar;
__u8 flags;
__u16 tslice;
__u16 phys_cpus;
__u16 this_part;
__u64 curtod;
} __attribute__ ((packed));
struct x_info_blk_hdr {
__u8 npar;
__u8 flags;
__u16 tslice;
__u16 phys_cpus;
__u16 this_part;
__u64 curtod1;
__u64 curtod2;
char reserved[40];
} __attribute__ ((packed));
static inline int info_blk_hdr__size(enum diag204_format type)
{
if (type == INFO_SIMPLE)
return sizeof(struct info_blk_hdr);
else /* INFO_EXT */
return sizeof(struct x_info_blk_hdr);
if (type == DIAG204_INFO_SIMPLE)
return sizeof(struct diag204_info_blk_hdr);
else /* DIAG204_INFO_EXT */
return sizeof(struct diag204_x_info_blk_hdr);
}
static inline __u8 info_blk_hdr__npar(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct info_blk_hdr *)hdr)->npar;
else /* INFO_EXT */
return ((struct x_info_blk_hdr *)hdr)->npar;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_info_blk_hdr *)hdr)->npar;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_info_blk_hdr *)hdr)->npar;
}
static inline __u8 info_blk_hdr__flags(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct info_blk_hdr *)hdr)->flags;
else /* INFO_EXT */
return ((struct x_info_blk_hdr *)hdr)->flags;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_info_blk_hdr *)hdr)->flags;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_info_blk_hdr *)hdr)->flags;
}
static inline __u16 info_blk_hdr__pcpus(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct info_blk_hdr *)hdr)->phys_cpus;
else /* INFO_EXT */
return ((struct x_info_blk_hdr *)hdr)->phys_cpus;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_info_blk_hdr *)hdr)->phys_cpus;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_info_blk_hdr *)hdr)->phys_cpus;
}
/* Partition header */
struct part_hdr {
__u8 pn;
__u8 cpus;
char reserved[6];
char part_name[LPAR_NAME_LEN];
} __attribute__ ((packed));
struct x_part_hdr {
__u8 pn;
__u8 cpus;
__u8 rcpus;
__u8 pflag;
__u32 mlu;
char part_name[LPAR_NAME_LEN];
char lpc_name[8];
char os_name[8];
__u64 online_cs;
__u64 online_es;
__u8 upid;
char reserved1[3];
__u32 group_mlu;
char group_name[8];
char reserved2[32];
} __attribute__ ((packed));
static inline int part_hdr__size(enum diag204_format type)
{
if (type == INFO_SIMPLE)
return sizeof(struct part_hdr);
else /* INFO_EXT */
return sizeof(struct x_part_hdr);
if (type == DIAG204_INFO_SIMPLE)
return sizeof(struct diag204_part_hdr);
else /* DIAG204_INFO_EXT */
return sizeof(struct diag204_x_part_hdr);
}
static inline __u8 part_hdr__rcpus(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct part_hdr *)hdr)->cpus;
else /* INFO_EXT */
return ((struct x_part_hdr *)hdr)->rcpus;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_part_hdr *)hdr)->cpus;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_part_hdr *)hdr)->rcpus;
}
static inline void part_hdr__part_name(enum diag204_format type, void *hdr,
char *name)
{
if (type == INFO_SIMPLE)
memcpy(name, ((struct part_hdr *)hdr)->part_name,
LPAR_NAME_LEN);
else /* INFO_EXT */
memcpy(name, ((struct x_part_hdr *)hdr)->part_name,
LPAR_NAME_LEN);
EBCASC(name, LPAR_NAME_LEN);
name[LPAR_NAME_LEN] = 0;
if (type == DIAG204_INFO_SIMPLE)
memcpy(name, ((struct diag204_part_hdr *)hdr)->part_name,
DIAG204_LPAR_NAME_LEN);
else /* DIAG204_INFO_EXT */
memcpy(name, ((struct diag204_x_part_hdr *)hdr)->part_name,
DIAG204_LPAR_NAME_LEN);
EBCASC(name, DIAG204_LPAR_NAME_LEN);
name[DIAG204_LPAR_NAME_LEN] = 0;
strim(name);
}
struct cpu_info {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
__u8 cflag;
__u16 weight;
__u64 acc_time;
__u64 lp_time;
} __attribute__ ((packed));
struct x_cpu_info {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
__u8 cflag;
__u16 weight;
__u64 acc_time;
__u64 lp_time;
__u16 min_weight;
__u16 cur_weight;
__u16 max_weight;
char reseved2[2];
__u64 online_time;
__u64 wait_time;
__u32 pma_weight;
__u32 polar_weight;
char reserved3[40];
} __attribute__ ((packed));
/* CPU info block */
static inline int cpu_info__size(enum diag204_format type)
{
if (type == INFO_SIMPLE)
return sizeof(struct cpu_info);
else /* INFO_EXT */
return sizeof(struct x_cpu_info);
if (type == DIAG204_INFO_SIMPLE)
return sizeof(struct diag204_cpu_info);
else /* DIAG204_INFO_EXT */
return sizeof(struct diag204_x_cpu_info);
}
static inline __u8 cpu_info__ctidx(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct cpu_info *)hdr)->ctidx;
else /* INFO_EXT */
return ((struct x_cpu_info *)hdr)->ctidx;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_cpu_info *)hdr)->ctidx;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_cpu_info *)hdr)->ctidx;
}
static inline __u16 cpu_info__cpu_addr(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct cpu_info *)hdr)->cpu_addr;
else /* INFO_EXT */
return ((struct x_cpu_info *)hdr)->cpu_addr;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_cpu_info *)hdr)->cpu_addr;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_cpu_info *)hdr)->cpu_addr;
}
static inline __u64 cpu_info__acc_time(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct cpu_info *)hdr)->acc_time;
else /* INFO_EXT */
return ((struct x_cpu_info *)hdr)->acc_time;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_cpu_info *)hdr)->acc_time;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_cpu_info *)hdr)->acc_time;
}
static inline __u64 cpu_info__lp_time(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct cpu_info *)hdr)->lp_time;
else /* INFO_EXT */
return ((struct x_cpu_info *)hdr)->lp_time;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_cpu_info *)hdr)->lp_time;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_cpu_info *)hdr)->lp_time;
}
static inline __u64 cpu_info__online_time(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
if (type == DIAG204_INFO_SIMPLE)
return 0; /* online_time not available in simple info */
else /* INFO_EXT */
return ((struct x_cpu_info *)hdr)->online_time;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_cpu_info *)hdr)->online_time;
}
/* Physical header */
struct phys_hdr {
char reserved1[1];
__u8 cpus;
char reserved2[6];
char mgm_name[8];
} __attribute__ ((packed));
struct x_phys_hdr {
char reserved1[1];
__u8 cpus;
char reserved2[6];
char mgm_name[8];
char reserved3[80];
} __attribute__ ((packed));
static inline int phys_hdr__size(enum diag204_format type)
{
if (type == INFO_SIMPLE)
return sizeof(struct phys_hdr);
else /* INFO_EXT */
return sizeof(struct x_phys_hdr);
if (type == DIAG204_INFO_SIMPLE)
return sizeof(struct diag204_phys_hdr);
else /* DIAG204_INFO_EXT */
return sizeof(struct diag204_x_phys_hdr);
}
static inline __u8 phys_hdr__cpus(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct phys_hdr *)hdr)->cpus;
else /* INFO_EXT */
return ((struct x_phys_hdr *)hdr)->cpus;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_phys_hdr *)hdr)->cpus;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_phys_hdr *)hdr)->cpus;
}
/* Physical CPU info block */
struct phys_cpu {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
char reserved2[3];
__u64 mgm_time;
char reserved3[8];
} __attribute__ ((packed));
struct x_phys_cpu {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
char reserved2[3];
__u64 mgm_time;
char reserved3[80];
} __attribute__ ((packed));
static inline int phys_cpu__size(enum diag204_format type)
{
if (type == INFO_SIMPLE)
return sizeof(struct phys_cpu);
else /* INFO_EXT */
return sizeof(struct x_phys_cpu);
if (type == DIAG204_INFO_SIMPLE)
return sizeof(struct diag204_phys_cpu);
else /* DIAG204_INFO_EXT */
return sizeof(struct diag204_x_phys_cpu);
}
static inline __u16 phys_cpu__cpu_addr(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct phys_cpu *)hdr)->cpu_addr;
else /* INFO_EXT */
return ((struct x_phys_cpu *)hdr)->cpu_addr;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_phys_cpu *)hdr)->cpu_addr;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_phys_cpu *)hdr)->cpu_addr;
}
static inline __u64 phys_cpu__mgm_time(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct phys_cpu *)hdr)->mgm_time;
else /* INFO_EXT */
return ((struct x_phys_cpu *)hdr)->mgm_time;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_phys_cpu *)hdr)->mgm_time;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_phys_cpu *)hdr)->mgm_time;
}
static inline __u64 phys_cpu__ctidx(enum diag204_format type, void *hdr)
{
if (type == INFO_SIMPLE)
return ((struct phys_cpu *)hdr)->ctidx;
else /* INFO_EXT */
return ((struct x_phys_cpu *)hdr)->ctidx;
if (type == DIAG204_INFO_SIMPLE)
return ((struct diag204_phys_cpu *)hdr)->ctidx;
else /* DIAG204_INFO_EXT */
return ((struct diag204_x_phys_cpu *)hdr)->ctidx;
}
/* Diagnose 204 functions */
static inline int __diag204(unsigned long *subcode, unsigned long size, void *addr)
{
register unsigned long _subcode asm("0") = *subcode;
register unsigned long _size asm("1") = size;
asm volatile(
" diag %2,%0,0x204\n"
"0: nopr %%r7\n"
EX_TABLE(0b,0b)
: "+d" (_subcode), "+d" (_size) : "d" (addr) : "memory");
*subcode = _subcode;
return _size;
}
static int diag204(unsigned long subcode, unsigned long size, void *addr)
{
diag_stat_inc(DIAG_STAT_X204);
size = __diag204(&subcode, size, addr);
if (subcode)
return -1;
return size;
}
/*
* For the old diag subcode 4 with simple data format we have to use real
* memory. If we use subcode 6 or 7 with extended data format, we can (and
@ -411,12 +261,12 @@ static void *diag204_get_buffer(enum diag204_format fmt, int *pages)
*pages = diag204_buf_pages;
return diag204_buf;
}
if (fmt == INFO_SIMPLE) {
if (fmt == DIAG204_INFO_SIMPLE) {
*pages = 1;
return diag204_alloc_rbuf();
} else {/* INFO_EXT */
*pages = diag204((unsigned long)SUBC_RSI |
(unsigned long)INFO_EXT, 0, NULL);
} else {/* DIAG204_INFO_EXT */
*pages = diag204((unsigned long)DIAG204_SUBC_RSI |
(unsigned long)DIAG204_INFO_EXT, 0, NULL);
if (*pages <= 0)
return ERR_PTR(-ENOSYS);
else
@ -443,18 +293,18 @@ static int diag204_probe(void)
void *buf;
int pages, rc;
buf = diag204_get_buffer(INFO_EXT, &pages);
buf = diag204_get_buffer(DIAG204_INFO_EXT, &pages);
if (!IS_ERR(buf)) {
if (diag204((unsigned long)SUBC_STIB7 |
(unsigned long)INFO_EXT, pages, buf) >= 0) {
diag204_store_sc = SUBC_STIB7;
diag204_info_type = INFO_EXT;
if (diag204((unsigned long)DIAG204_SUBC_STIB7 |
(unsigned long)DIAG204_INFO_EXT, pages, buf) >= 0) {
diag204_store_sc = DIAG204_SUBC_STIB7;
diag204_info_type = DIAG204_INFO_EXT;
goto out;
}
if (diag204((unsigned long)SUBC_STIB6 |
(unsigned long)INFO_EXT, pages, buf) >= 0) {
diag204_store_sc = SUBC_STIB6;
diag204_info_type = INFO_EXT;
if (diag204((unsigned long)DIAG204_SUBC_STIB6 |
(unsigned long)DIAG204_INFO_EXT, pages, buf) >= 0) {
diag204_store_sc = DIAG204_SUBC_STIB6;
diag204_info_type = DIAG204_INFO_EXT;
goto out;
}
diag204_free_buffer();
@ -462,15 +312,15 @@ static int diag204_probe(void)
/* subcodes 6 and 7 failed, now try subcode 4 */
buf = diag204_get_buffer(INFO_SIMPLE, &pages);
buf = diag204_get_buffer(DIAG204_INFO_SIMPLE, &pages);
if (IS_ERR(buf)) {
rc = PTR_ERR(buf);
goto fail_alloc;
}
if (diag204((unsigned long)SUBC_STIB4 |
(unsigned long)INFO_SIMPLE, pages, buf) >= 0) {
diag204_store_sc = SUBC_STIB4;
diag204_info_type = INFO_SIMPLE;
if (diag204((unsigned long)DIAG204_SUBC_STIB4 |
(unsigned long)DIAG204_INFO_SIMPLE, pages, buf) >= 0) {
diag204_store_sc = DIAG204_SUBC_STIB4;
diag204_info_type = DIAG204_INFO_SIMPLE;
goto out;
} else {
rc = -ENOSYS;
@ -510,20 +360,6 @@ static void *diag204_store(void)
/* Diagnose 224 functions */
static int diag224(void *ptr)
{
int rc = -EOPNOTSUPP;
diag_stat_inc(DIAG_STAT_X224);
asm volatile(
" diag %1,%2,0x224\n"
"0: lhi %0,0x0\n"
"1:\n"
EX_TABLE(0b,1b)
: "+d" (rc) :"d" (0), "d" (ptr) : "memory");
return rc;
}
static int diag224_get_name_table(void)
{
/* memory must be below 2GB */
@ -545,9 +381,9 @@ static void diag224_delete_name_table(void)
static int diag224_idx2name(int index, char *name)
{
memcpy(name, diag224_cpu_names + ((index + 1) * CPU_NAME_LEN),
CPU_NAME_LEN);
name[CPU_NAME_LEN] = 0;
memcpy(name, diag224_cpu_names + ((index + 1) * DIAG204_CPU_NAME_LEN),
DIAG204_CPU_NAME_LEN);
name[DIAG204_CPU_NAME_LEN] = 0;
strim(name);
return 0;
}
@ -603,7 +439,7 @@ __init int hypfs_diag_init(void)
pr_err("The hardware system does not support hypfs\n");
return -ENODATA;
}
if (diag204_info_type == INFO_EXT) {
if (diag204_info_type == DIAG204_INFO_EXT) {
rc = hypfs_dbfs_create_file(&dbfs_file_d204);
if (rc)
return rc;
@ -651,7 +487,7 @@ static int hypfs_create_cpu_files(struct dentry *cpus_dir, void *cpu_info)
cpu_info__lp_time(diag204_info_type, cpu_info));
if (IS_ERR(rc))
return PTR_ERR(rc);
if (diag204_info_type == INFO_EXT) {
if (diag204_info_type == DIAG204_INFO_EXT) {
rc = hypfs_create_u64(cpu_dir, "onlinetime",
cpu_info__online_time(diag204_info_type,
cpu_info));
@ -667,12 +503,12 @@ static void *hypfs_create_lpar_files(struct dentry *systems_dir, void *part_hdr)
{
struct dentry *cpus_dir;
struct dentry *lpar_dir;
char lpar_name[LPAR_NAME_LEN + 1];
char lpar_name[DIAG204_LPAR_NAME_LEN + 1];
void *cpu_info;
int i;
part_hdr__part_name(diag204_info_type, part_hdr, lpar_name);
lpar_name[LPAR_NAME_LEN] = 0;
lpar_name[DIAG204_LPAR_NAME_LEN] = 0;
lpar_dir = hypfs_mkdir(systems_dir, lpar_name);
if (IS_ERR(lpar_dir))
return lpar_dir;
@ -755,7 +591,8 @@ int hypfs_diag_create_files(struct dentry *root)
goto err_out;
}
}
if (info_blk_hdr__flags(diag204_info_type, time_hdr) & LPAR_PHYS_FLG) {
if (info_blk_hdr__flags(diag204_info_type, time_hdr) &
DIAG204_LPAR_PHYS_FLG) {
ptr = hypfs_create_phys_files(root, part_hdr);
if (IS_ERR(ptr)) {
rc = PTR_ERR(ptr);

10
arch/s390/include/asm/cpacf.h
View File

@ -20,6 +20,9 @@
#define CPACF_KMC 0xb92f /* MSA */
#define CPACF_KIMD 0xb93e /* MSA */
#define CPACF_KLMD 0xb93f /* MSA */
#define CPACF_PCKMO 0xb928 /* MSA3 */
#define CPACF_KMF 0xb92a /* MSA4 */
#define CPACF_KMO 0xb92b /* MSA4 */
#define CPACF_PCC 0xb92c /* MSA4 */
#define CPACF_KMCTR 0xb92d /* MSA4 */
#define CPACF_PPNO 0xb93c /* MSA5 */
@ -136,6 +139,7 @@ static inline void __cpacf_query(unsigned int opcode, unsigned char *status)
register unsigned long r1 asm("1") = (unsigned long) status;
asm volatile(
" spm 0\n" /* pckmo doesn't change the cc */
/* Parameter registers are ignored, but may not be 0 */
"0: .insn rrf,%[opc] << 16,2,2,2,0\n"
" brc 1,0b\n" /* handle partial completion */
@ -157,6 +161,12 @@ static inline int cpacf_query(unsigned int opcode, unsigned int func)
if (!test_facility(17)) /* check for MSA */
return 0;
break;
case CPACF_PCKMO:
if (!test_facility(76)) /* check for MSA3 */
return 0;
break;
case CPACF_KMF:
case CPACF_KMO:
case CPACF_PCC:
case CPACF_KMCTR:
if (!test_facility(77)) /* check for MSA4 */

149
arch/s390/include/asm/diag.h
View File

@ -78,4 +78,153 @@ struct diag210 {
extern int diag210(struct diag210 *addr);
/* bit is set in flags, when physical cpu info is included in diag 204 data */
#define DIAG204_LPAR_PHYS_FLG 0x80
#define DIAG204_LPAR_NAME_LEN 8 /* lpar name len in diag 204 data */
#define DIAG204_CPU_NAME_LEN 16 /* type name len of cpus in diag224 name table */
/* diag 204 subcodes */
enum diag204_sc {
DIAG204_SUBC_STIB4 = 4,
DIAG204_SUBC_RSI = 5,
DIAG204_SUBC_STIB6 = 6,
DIAG204_SUBC_STIB7 = 7
};
/* The two available diag 204 data formats */
enum diag204_format {
DIAG204_INFO_SIMPLE = 0,
DIAG204_INFO_EXT = 0x00010000
};
enum diag204_cpu_flags {
DIAG204_CPU_ONLINE = 0x20,
DIAG204_CPU_CAPPED = 0x40,
};
struct diag204_info_blk_hdr {
__u8 npar;
__u8 flags;
__u16 tslice;
__u16 phys_cpus;
__u16 this_part;
__u64 curtod;
} __packed;
struct diag204_x_info_blk_hdr {
__u8 npar;
__u8 flags;
__u16 tslice;
__u16 phys_cpus;
__u16 this_part;
__u64 curtod1;
__u64 curtod2;
char reserved[40];
} __packed;
struct diag204_part_hdr {
__u8 pn;
__u8 cpus;
char reserved[6];
char part_name[DIAG204_LPAR_NAME_LEN];
} __packed;
struct diag204_x_part_hdr {
__u8 pn;
__u8 cpus;
__u8 rcpus;
__u8 pflag;
__u32 mlu;
char part_name[DIAG204_LPAR_NAME_LEN];
char lpc_name[8];
char os_name[8];
__u64 online_cs;
__u64 online_es;
__u8 upid;
__u8 reserved:3;
__u8 mtid:5;
char reserved1[2];
__u32 group_mlu;
char group_name[8];
char hardware_group_name[8];
char reserved2[24];
} __packed;
struct diag204_cpu_info {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
__u8 cflag;
__u16 weight;
__u64 acc_time;
__u64 lp_time;
} __packed;
struct diag204_x_cpu_info {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
__u8 cflag;
__u16 weight;
__u64 acc_time;
__u64 lp_time;
__u16 min_weight;
__u16 cur_weight;
__u16 max_weight;
char reseved2[2];
__u64 online_time;
__u64 wait_time;
__u32 pma_weight;
__u32 polar_weight;
__u32 cpu_type_cap;
__u32 group_cpu_type_cap;
char reserved3[32];
} __packed;
struct diag204_phys_hdr {
char reserved1[1];
__u8 cpus;
char reserved2[6];
char mgm_name[8];
} __packed;
struct diag204_x_phys_hdr {
char reserved1[1];
__u8 cpus;
char reserved2[6];
char mgm_name[8];
char reserved3[80];
} __packed;
struct diag204_phys_cpu {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
char reserved2[3];
__u64 mgm_time;
char reserved3[8];
} __packed;
struct diag204_x_phys_cpu {
__u16 cpu_addr;
char reserved1[2];
__u8 ctidx;
char reserved2[1];
__u16 weight;
__u64 mgm_time;
char reserved3[80];
} __packed;
struct diag204_x_part_block {
struct diag204_x_part_hdr hdr;
struct diag204_x_cpu_info cpus[];
} __packed;
struct diag204_x_phys_block {
struct diag204_x_phys_hdr hdr;
struct diag204_x_phys_cpu cpus[];
} __packed;
int diag204(unsigned long subcode, unsigned long size, void *addr);
int diag224(void *ptr);
#endif /* _ASM_S390_DIAG_H */

82
arch/s390/include/asm/gmap.h
View File

@ -10,14 +10,25 @@
/**
* struct gmap_struct - guest address space
* @list: list head for the mm->context gmap list
* @crst_list: list of all crst tables used in the guest address space
* @mm: pointer to the parent mm_struct
* @guest_to_host: radix tree with guest to host address translation
* @host_to_guest: radix tree with pointer to segment table entries
* @guest_table_lock: spinlock to protect all entries in the guest page table
* @ref_count: reference counter for the gmap structure
* @table: pointer to the page directory
* @asce: address space control element for gmap page table
* @pfault_enabled: defines if pfaults are applicable for the guest
* @host_to_rmap: radix tree with gmap_rmap lists
* @children: list of shadow gmap structures
* @pt_list: list of all page tables used in the shadow guest address space
* @shadow_lock: spinlock to protect the shadow gmap list
* @parent: pointer to the parent gmap for shadow guest address spaces
* @orig_asce: ASCE for which the shadow page table has been created
* @edat_level: edat level to be used for the shadow translation
* @removed: flag to indicate if a shadow guest address space has been removed
* @initialized: flag to indicate if a shadow guest address space can be used
*/
struct gmap {
struct list_head list;
@ -26,26 +37,64 @@ struct gmap {
struct radix_tree_root guest_to_host;
struct radix_tree_root host_to_guest;
spinlock_t guest_table_lock;
atomic_t ref_count;
unsigned long *table;
unsigned long asce;
unsigned long asce_end;
void *private;
bool pfault_enabled;
/* Additional data for shadow guest address spaces */
struct radix_tree_root host_to_rmap;
struct list_head children;
struct list_head pt_list;
spinlock_t shadow_lock;
struct gmap *parent;
unsigned long orig_asce;
int edat_level;
bool removed;
bool initialized;
};
/**
* struct gmap_rmap - reverse mapping for shadow page table entries
* @next: pointer to next rmap in the list
* @raddr: virtual rmap address in the shadow guest address space
*/
struct gmap_rmap {
struct gmap_rmap *next;
unsigned long raddr;
};
#define gmap_for_each_rmap(pos, head) \
for (pos = (head); pos; pos = pos->next)
#define gmap_for_each_rmap_safe(pos, n, head) \
for (pos = (head); n = pos ? pos->next : NULL, pos; pos = n)
/**
* struct gmap_notifier - notify function block for page invalidation
* @notifier_call: address of callback function
*/
struct gmap_notifier {
struct list_head list;
void (*notifier_call)(struct gmap *gmap, unsigned long gaddr);
struct rcu_head rcu;
void (*notifier_call)(struct gmap *gmap, unsigned long start,
unsigned long end);
};
struct gmap *gmap_alloc(struct mm_struct *mm, unsigned long limit);
void gmap_free(struct gmap *gmap);
static inline int gmap_is_shadow(struct gmap *gmap)
{
return !!gmap->parent;
}
struct gmap *gmap_create(struct mm_struct *mm, unsigned long limit);
void gmap_remove(struct gmap *gmap);
struct gmap *gmap_get(struct gmap *gmap);
void gmap_put(struct gmap *gmap);
void gmap_enable(struct gmap *gmap);
void gmap_disable(struct gmap *gmap);
struct gmap *gmap_get_enabled(void);
int gmap_map_segment(struct gmap *gmap, unsigned long from,
unsigned long to, unsigned long len);
int gmap_unmap_segment(struct gmap *gmap, unsigned long to, unsigned long len);
@ -57,8 +106,29 @@ void gmap_discard(struct gmap *, unsigned long from, unsigned long to);
void __gmap_zap(struct gmap *, unsigned long gaddr);
void gmap_unlink(struct mm_struct *, unsigned long *table, unsigned long vmaddr);
void gmap_register_ipte_notifier(struct gmap_notifier *);
void gmap_unregister_ipte_notifier(struct gmap_notifier *);
int gmap_ipte_notify(struct gmap *, unsigned long start, unsigned long len);
int gmap_read_table(struct gmap *gmap, unsigned long gaddr, unsigned long *val);
struct gmap *gmap_shadow(struct gmap *parent, unsigned long asce,
int edat_level);
int gmap_shadow_valid(struct gmap *sg, unsigned long asce, int edat_level);
int gmap_shadow_r2t(struct gmap *sg, unsigned long saddr, unsigned long r2t,
int fake);
int gmap_shadow_r3t(struct gmap *sg, unsigned long saddr, unsigned long r3t,
int fake);
int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt,
int fake);
int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt,
int fake);
int gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long saddr,
unsigned long *pgt, int *dat_protection, int *fake);
int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte);
void gmap_register_pte_notifier(struct gmap_notifier *);
void gmap_unregister_pte_notifier(struct gmap_notifier *);
void gmap_pte_notify(struct mm_struct *, unsigned long addr, pte_t *,
unsigned long bits);
int gmap_mprotect_notify(struct gmap *, unsigned long start,
unsigned long len, int prot);
#endif /* _ASM_S390_GMAP_H */

33
arch/s390/include/asm/kvm_host.h
View File

@ -43,6 +43,7 @@
/* s390-specific vcpu->requests bit members */
#define KVM_REQ_ENABLE_IBS 8
#define KVM_REQ_DISABLE_IBS 9
#define KVM_REQ_ICPT_OPEREXC 10
#define SIGP_CTRL_C 0x80
#define SIGP_CTRL_SCN_MASK 0x3f
@ -145,7 +146,7 @@ struct kvm_s390_sie_block {
__u64 cputm; /* 0x0028 */
__u64 ckc; /* 0x0030 */
__u64 epoch; /* 0x0038 */
__u8 reserved40[4]; /* 0x0040 */
__u32 svcc; /* 0x0040 */
#define LCTL_CR0 0x8000
#define LCTL_CR6 0x0200
#define LCTL_CR9 0x0040
@ -154,6 +155,7 @@ struct kvm_s390_sie_block {
#define LCTL_CR14 0x0002
__u16 lctl; /* 0x0044 */
__s16 icpua; /* 0x0046 */
#define ICTL_OPEREXC 0x80000000
#define ICTL_PINT 0x20000000
#define ICTL_LPSW 0x00400000
#define ICTL_STCTL 0x00040000
@ -166,6 +168,9 @@ struct kvm_s390_sie_block {
#define ICPT_INST 0x04
#define ICPT_PROGI 0x08
#define ICPT_INSTPROGI 0x0C
#define ICPT_EXTINT 0x14
#define ICPT_VALIDITY 0x20
#define ICPT_STOP 0x28
#define ICPT_OPEREXC 0x2C
#define ICPT_PARTEXEC 0x38
#define ICPT_IOINST 0x40
@ -185,7 +190,9 @@ struct kvm_s390_sie_block {
__u32 scaol; /* 0x0064 */
__u8 reserved68[4]; /* 0x0068 */
__u32 todpr; /* 0x006c */
__u8 reserved70[32]; /* 0x0070 */
__u8 reserved70[16]; /* 0x0070 */
__u64 mso; /* 0x0080 */
__u64 msl; /* 0x0088 */
psw_t gpsw; /* 0x0090 */
__u64 gg14; /* 0x00a0 */
__u64 gg15; /* 0x00a8 */
@ -223,7 +230,7 @@ struct kvm_s390_sie_block {
__u8 reserved1e6[2]; /* 0x01e6 */
__u64 itdba; /* 0x01e8 */
__u64 riccbd; /* 0x01f0 */
__u8 reserved1f8[8]; /* 0x01f8 */
__u64 gvrd; /* 0x01f8 */
} __attribute__((packed));
struct kvm_s390_itdb {
@ -256,6 +263,7 @@ struct kvm_vcpu_stat {
u32 instruction_stctg;
u32 exit_program_interruption;
u32 exit_instr_and_program;
u32 exit_operation_exception;
u32 deliver_external_call;
u32 deliver_emergency_signal;
u32 deliver_service_signal;
@ -278,7 +286,9 @@ struct kvm_vcpu_stat {
u32 instruction_stsi;
u32 instruction_stfl;
u32 instruction_tprot;
u32 instruction_sie;
u32 instruction_essa;
u32 instruction_sthyi;
u32 instruction_sigp_sense;
u32 instruction_sigp_sense_running;
u32 instruction_sigp_external_call;
@ -541,12 +551,16 @@ struct kvm_guestdbg_info_arch {
struct kvm_vcpu_arch {
struct kvm_s390_sie_block *sie_block;
/* if vsie is active, currently executed shadow sie control block */
struct kvm_s390_sie_block *vsie_block;
unsigned int host_acrs[NUM_ACRS];
struct fpu host_fpregs;
struct kvm_s390_local_interrupt local_int;
struct hrtimer ckc_timer;
struct kvm_s390_pgm_info pgm;
struct gmap *gmap;
/* backup location for the currently enabled gmap when scheduled out */
struct gmap *enabled_gmap;
struct kvm_guestdbg_info_arch guestdbg;
unsigned long pfault_token;
unsigned long pfault_select;
@ -631,6 +645,14 @@ struct sie_page2 {
u8 reserved900[0x1000 - 0x900]; /* 0x0900 */
} __packed;
struct kvm_s390_vsie {
struct mutex mutex;
struct radix_tree_root addr_to_page;
int page_count;
int next;
struct page *pages[KVM_MAX_VCPUS];
};
struct kvm_arch{
void *sca;
int use_esca;
@ -646,15 +668,20 @@ struct kvm_arch{
int user_cpu_state_ctrl;
int user_sigp;
int user_stsi;
int user_instr0;
struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS];
wait_queue_head_t ipte_wq;
int ipte_lock_count;
struct mutex ipte_mutex;
struct ratelimit_state sthyi_limit;
spinlock_t start_stop_lock;
struct sie_page2 *sie_page2;
struct kvm_s390_cpu_model model;
struct kvm_s390_crypto crypto;
struct kvm_s390_vsie vsie;
u64 epoch;
/* subset of available cpu features enabled by user space */
DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
};
#define KVM_HVA_ERR_BAD (-1UL)

11
arch/s390/include/asm/mmu.h
View File

@ -8,8 +8,9 @@ typedef struct {
cpumask_t cpu_attach_mask;
atomic_t flush_count;
unsigned int flush_mm;
spinlock_t list_lock;
spinlock_t pgtable_lock;
struct list_head pgtable_list;
spinlock_t gmap_lock;
struct list_head gmap_list;
unsigned long asce;
unsigned long asce_limit;
@ -22,9 +23,11 @@ typedef struct {
unsigned int use_skey:1;
} mm_context_t;
#define INIT_MM_CONTEXT(name) \
.context.list_lock = __SPIN_LOCK_UNLOCKED(name.context.list_lock), \
.context.pgtable_list = LIST_HEAD_INIT(name.context.pgtable_list), \
#define INIT_MM_CONTEXT(name) \
.context.pgtable_lock = \
__SPIN_LOCK_UNLOCKED(name.context.pgtable_lock), \
.context.pgtable_list = LIST_HEAD_INIT(name.context.pgtable_list), \
.context.gmap_lock = __SPIN_LOCK_UNLOCKED(name.context.gmap_lock), \
.context.gmap_list = LIST_HEAD_INIT(name.context.gmap_list),
static inline int tprot(unsigned long addr)

3
arch/s390/include/asm/mmu_context.h
View File

@ -15,8 +15,9 @@
static inline int init_new_context(struct task_struct *tsk,
struct mm_struct *mm)
{
spin_lock_init(&mm->context.list_lock);
spin_lock_init(&mm->context.pgtable_lock);
INIT_LIST_HEAD(&mm->context.pgtable_list);
spin_lock_init(&mm->context.gmap_lock);
INIT_LIST_HEAD(&mm->context.gmap_list);
cpumask_clear(&mm->context.cpu_attach_mask);
atomic_set(&mm->context.flush_count, 0);

9
arch/s390/include/asm/page.h
View File

@ -111,13 +111,14 @@ static inline unsigned char page_get_storage_key(unsigned long addr)
static inline int page_reset_referenced(unsigned long addr)
{
unsigned int ipm;
int cc;
asm volatile(
" rrbe 0,%1\n"
" ipm %0\n"
: "=d" (ipm) : "a" (addr) : "cc");
return !!(ipm & 0x20000000);
" srl %0,28\n"
: "=d" (cc) : "a" (addr) : "cc");
return cc;
}
/* Bits int the storage key */
@ -148,6 +149,8 @@ static inline int devmem_is_allowed(unsigned long pfn)
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
#define page_to_phys(page) (page_to_pfn(page) << PAGE_SHIFT)
#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
#define pfn_to_virt(pfn) __va((pfn) << PAGE_SHIFT)
#define page_to_virt(page) pfn_to_virt(page_to_pfn(page))
#define VM_DATA_DEFAULT_FLAGS (VM_READ | VM_WRITE | \
VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC)

2
arch/s390/include/asm/pgalloc.h
View File

@ -19,8 +19,10 @@ unsigned long *crst_table_alloc(struct mm_struct *);
void crst_table_free(struct mm_struct *, unsigned long *);
unsigned long *page_table_alloc(struct mm_struct *);
struct page *page_table_alloc_pgste(struct mm_struct *mm);
void page_table_free(struct mm_struct *, unsigned long *);
void page_table_free_rcu(struct mmu_gather *, unsigned long *, unsigned long);
void page_table_free_pgste(struct page *page);
extern int page_table_allocate_pgste;
static inline void clear_table(unsigned long *s, unsigned long val, size_t n)

17
arch/s390/include/asm/pgtable.h
View File

@ -277,6 +277,7 @@ static inline int is_module_addr(void *addr)
/* Bits in the region table entry */
#define _REGION_ENTRY_ORIGIN ~0xfffUL/* region/segment table origin */
#define _REGION_ENTRY_PROTECT 0x200 /* region protection bit */
#define _REGION_ENTRY_OFFSET 0xc0 /* region table offset */
#define _REGION_ENTRY_INVALID 0x20 /* invalid region table entry */
#define _REGION_ENTRY_TYPE_MASK 0x0c /* region/segment table type mask */
#define _REGION_ENTRY_TYPE_R1 0x0c /* region first table type */
@ -364,6 +365,7 @@ static inline int is_module_addr(void *addr)
#define PGSTE_GC_BIT 0x0002000000000000UL
#define PGSTE_UC_BIT 0x0000800000000000UL /* user dirty (migration) */
#define PGSTE_IN_BIT 0x0000400000000000UL /* IPTE notify bit */
#define PGSTE_VSIE_BIT 0x0000200000000000UL /* ref'd in a shadow table */
/* Guest Page State used for virtualization */
#define _PGSTE_GPS_ZERO 0x0000000080000000UL
@ -1002,15 +1004,26 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma,
void ptep_set_pte_at(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t entry);
void ptep_set_notify(struct mm_struct *mm, unsigned long addr, pte_t *ptep);
void ptep_notify(struct mm_struct *mm, unsigned long addr, pte_t *ptep);
void ptep_notify(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, unsigned long bits);
int ptep_force_prot(struct mm_struct *mm, unsigned long gaddr,
pte_t *ptep, int prot, unsigned long bit);
void ptep_zap_unused(struct mm_struct *mm, unsigned long addr,
pte_t *ptep , int reset);
void ptep_zap_key(struct mm_struct *mm, unsigned long addr, pte_t *ptep);
int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr,
pte_t *sptep, pte_t *tptep, pte_t pte);
void ptep_unshadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *ptep);
bool test_and_clear_guest_dirty(struct mm_struct *mm, unsigned long address);
int set_guest_storage_key(struct mm_struct *mm, unsigned long addr,
unsigned char key, bool nq);
unsigned char get_guest_storage_key(struct mm_struct *mm, unsigned long addr);
int cond_set_guest_storage_key(struct mm_struct *mm, unsigned long addr,
unsigned char key, unsigned char *oldkey,
bool nq, bool mr, bool mc);
int reset_guest_reference_bit(struct mm_struct *mm, unsigned long addr);
int get_guest_storage_key(struct mm_struct *mm, unsigned long addr,
unsigned char *key);
/*
* Certain architectures need to do special things when PTEs

2
arch/s390/include/asm/processor.h
View File

@ -112,6 +112,8 @@ struct thread_struct {
unsigned long ksp; /* kernel stack pointer */
mm_segment_t mm_segment;
unsigned long gmap_addr; /* address of last gmap fault. */
unsigned int gmap_write_flag; /* gmap fault write indication */
unsigned int gmap_int_code; /* int code of last gmap fault */
unsigned int gmap_pfault; /* signal of a pending guest pfault */
struct per_regs per_user; /* User specified PER registers */
struct per_event per_event; /* Cause of the last PER trap */

23
arch/s390/include/asm/sclp.h
View File

@ -32,12 +32,19 @@ struct sclp_core_entry {
u8 reserved0;
u8 : 4;
u8 sief2 : 1;
u8 : 3;
u8 : 3;
u8 skey : 1;
u8 : 2;
u8 : 2;
u8 gpere : 1;
u8 siif : 1;
u8 sigpif : 1;
u8 : 3;
u8 reserved2[10];
u8 reserved2[3];
u8 : 2;
u8 ib : 1;
u8 cei : 1;
u8 : 4;
u8 reserved3[6];
u8 type;
u8 reserved1;
} __attribute__((packed));
@ -59,6 +66,15 @@ struct sclp_info {
unsigned char has_hvs : 1;
unsigned char has_esca : 1;
unsigned char has_sief2 : 1;
unsigned char has_64bscao : 1;
unsigned char has_gpere : 1;
unsigned char has_cmma : 1;
unsigned char has_gsls : 1;
unsigned char has_ib : 1;
unsigned char has_cei : 1;
unsigned char has_pfmfi : 1;
unsigned char has_ibs : 1;
unsigned char has_skey : 1;
unsigned int ibc;
unsigned int mtid;
unsigned int mtid_cp;
@ -101,5 +117,6 @@ int memcpy_hsa_kernel(void *dest, unsigned long src, size_t count);
int memcpy_hsa_user(void __user *dest, unsigned long src, size_t count);
void sclp_early_detect(void);
void _sclp_print_early(const char *);
void sclp_ocf_cpc_name_copy(char *dst);
#endif /* _ASM_S390_SCLP_H */

41
arch/s390/include/uapi/asm/kvm.h
View File

@ -93,6 +93,47 @@ struct kvm_s390_vm_cpu_machine {
__u64 fac_list[256];
};
#define KVM_S390_VM_CPU_PROCESSOR_FEAT 2
#define KVM_S390_VM_CPU_MACHINE_FEAT 3
#define KVM_S390_VM_CPU_FEAT_NR_BITS 1024
#define KVM_S390_VM_CPU_FEAT_ESOP 0
#define KVM_S390_VM_CPU_FEAT_SIEF2 1
#define KVM_S390_VM_CPU_FEAT_64BSCAO 2
#define KVM_S390_VM_CPU_FEAT_SIIF 3
#define KVM_S390_VM_CPU_FEAT_GPERE 4
#define KVM_S390_VM_CPU_FEAT_GSLS 5
#define KVM_S390_VM_CPU_FEAT_IB 6
#define KVM_S390_VM_CPU_FEAT_CEI 7
#define KVM_S390_VM_CPU_FEAT_IBS 8
#define KVM_S390_VM_CPU_FEAT_SKEY 9
#define KVM_S390_VM_CPU_FEAT_CMMA 10
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};
#define KVM_S390_VM_CPU_PROCESSOR_SUBFUNC 4
#define KVM_S390_VM_CPU_MACHINE_SUBFUNC 5
/* for "test bit" instructions MSB 0 bit ordering, for "query" raw blocks */
struct kvm_s390_vm_cpu_subfunc {
__u8 plo[32]; /* always */
__u8 ptff[16]; /* with TOD-clock steering */
__u8 kmac[16]; /* with MSA */
__u8 kmc[16]; /* with MSA */
__u8 km[16]; /* with MSA */
__u8 kimd[16]; /* with MSA */
__u8 klmd[16]; /* with MSA */
__u8 pckmo[16]; /* with MSA3 */
__u8 kmctr[16]; /* with MSA4 */
__u8 kmf[16]; /* with MSA4 */
__u8 kmo[16]; /* with MSA4 */
__u8 pcc[16]; /* with MSA4 */
__u8 ppno[16]; /* with MSA5 */
__u8 reserved[1824];
};
/* kvm attributes for crypto */
#define KVM_S390_VM_CRYPTO_ENABLE_AES_KW 0
#define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1

1
arch/s390/include/uapi/asm/sie.h
View File

@ -140,6 +140,7 @@
exit_code_ipa0(0xB2, 0x4c, "TAR"), \
exit_code_ipa0(0xB2, 0x50, "CSP"), \
exit_code_ipa0(0xB2, 0x54, "MVPG"), \
exit_code_ipa0(0xB2, 0x56, "STHYI"), \
exit_code_ipa0(0xB2, 0x58, "BSG"), \
exit_code_ipa0(0xB2, 0x5a, "BSA"), \
exit_code_ipa0(0xB2, 0x5f, "CHSC"), \

39
arch/s390/kernel/diag.c
View File

@ -162,6 +162,30 @@ int diag14(unsigned long rx, unsigned long ry1, unsigned long subcode)
}
EXPORT_SYMBOL(diag14);
static inline int __diag204(unsigned long *subcode, unsigned long size, void *addr)
{
register unsigned long _subcode asm("0") = *subcode;
register unsigned long _size asm("1") = size;
asm volatile(
" diag %2,%0,0x204\n"
"0: nopr %%r7\n"
EX_TABLE(0b,0b)
: "+d" (_subcode), "+d" (_size) : "d" (addr) : "memory");
*subcode = _subcode;
return _size;
}
int diag204(unsigned long subcode, unsigned long size, void *addr)
{
diag_stat_inc(DIAG_STAT_X204);
size = __diag204(&subcode, size, addr);
if (subcode)
return -1;
return size;
}
EXPORT_SYMBOL(diag204);
/*
* Diagnose 210: Get information about a virtual device
*/
@ -196,3 +220,18 @@ int diag210(struct diag210 *addr)
return ccode;
}
EXPORT_SYMBOL(diag210);
int diag224(void *ptr)
{
int rc = -EOPNOTSUPP;
diag_stat_inc(DIAG_STAT_X224);
asm volatile(
" diag %1,%2,0x224\n"
"0: lhi %0,0x0\n"
"1:\n"
EX_TABLE(0b,1b)
: "+d" (rc) :"d" (0), "d" (ptr) : "memory");
return rc;
}
EXPORT_SYMBOL(diag224);

Some files were not shown because too many files have changed in this diff Show More