FreeBSD Refactor in Virtual Ghost

Q&A

Todos

  • Classify all code changes in SVA (combine paper and actual code)

Userspace

libc (malloc)

File: lib/libc/stdlib/malloc.c

  • disable sbrk() by undefine MALLOC_DSS macro. LLM: What is this mean?

    code

  • Add ghost_sbrk(intptr_t incr) and use it to replace sbrk. ghost_sbrk use secmemalloc when GHOSTING. secmemalloc(uintptr_t size) is implemented as an interrupt (number 0x7f.

    code

  • disable TLS. define ‘NO_TLS’ macro.

Kernel Structures

LLM: Will kernel’s errorneously change to these SVA related fields invalidate SVA protection???

pcpu

  • 5 More fields are added to kernel’s pcpu struct.
    code

thread struct

  • sva added 6 more elements:
    code

Interrupt

sys/sys/amd64/amd64/

  • trap.c

    • add void fr_sva_trap(unsigned trapno, void * trapAddr) to replace void trap(struct trapframe *frame)
      • enable interrupts;
      • convert SVA trap frame freebsd frame;
      • call trap_pfault_sva instead of trap_pfault
      • no fpu support in sva;
      • run ast(frame);
    • add trap_fault_sva
    • add trap_pfault_sva
    • void amd64_syscall(struct thread *td, int traced):
    • void sva_syscall(struct thread *td, int traced):
      • enable interrupt;
      • sva_syscall_trapframe(&localframe): convert sva trap frame to freebsd trap frame.
      • ast(&localframe)
    • void trap(struct trapframe *frame):
      • add checking on trap type, panic if not T_BPTFLT nor T_NMI:
        code
  • mp_machdep.c

    • init_secondary()
    • keep common_tss[cpu].tss_ist3 that is SVA IST configuration
      code
    • disable lidt($r_idt);
    • call sva_init_secondary() to initialize the processor.
    • call sva_load_cr0(cr0) instead of load_cr0(cr0).
  • machdep.c

    • setidt(idx,func,typ,dpl,ist):
    • check to disable changes to the 0x7f interrupt (SVA)
    • spurious_handler(unsigned intNum, sva_icontext *p): new func
    • hammer_time(u_int64_t modulep, u_int64_t physfree):
    • changed: KGSBASE set to pcpu[0] instead of 0: wrmsr(MSR_KGSBASE, (u_int64_t)pc)
    • added: initialize pcpu[i].svaIContext from sva_getCPUState(&common_tss[i]).
    • added: initial SVA VM on the primary processor: sva_init_primary();
    • changed: call SVA to initialize exception.
      • sva_register_general_exception(IDT_DE,fr_sva_trap) to replace setidt(IDT_DE,&IDTVEC(div));
      • sva_register_general_exception(IDT_OF, fr_sva_trap);
      • sva_register_general_exception(IDT_BR, fr_sva_trap);
      • sva_register_general_exception(IDT_UD, fr_sva_trap);
    • added: thread0.svaID = 0;
    • added: cpu_switch_sva(old,new,mtx) to replace cpu_switch() in cpu_switch.S.
  • exception.S

    • add SVAsyscall, call this during IDTVEC(fast_syscall)
    • Orignal fast_syscall:
    • add SVAsysret
    • add RETTARGET label.
  • locore.S & apic_vector.S & atpic_vector.S & cpu_switch.S: sva/cfi.h:

    • RETTARGET after every call: label of every return target.
    • RETQ to replace ret: first check label of return address, then return.
  • APIC sys/x86/x86/local_apic.c

    • use apic_isr_svax instead of apic_isrx;
    • use sva_reg: spurious_handler instead of setidt: spuriousint;
    • use sva_reg: lapic_handle_timer instead of setidt: timerint;
    • use sva_reg: lapic_handle_error instead of setidt: errorint;
    • add lapic_handle_intr_sva(int vector)
    • add apic_isr_sva1(unsigned int vector)
    • add apic_isr_sva2(..)
    • use lapic_handle_timer(int type) instead of lapic_handle_timer(struct trapframe *frame); // defined in include/apicvar.h
  • sys/kern/subr_trap.c:

    • ast(struct trapframe *framep): enable_intr() at beginning;

Process management

sys/sys/amd64/amd64/vm_machdep.c

  • add void kernel_thread_trampoline (struct thread * td): call fork_exit with td-> callout, and td->callarg;

    code

  • cpu_fork(…): init thread struct: .sva/.callout/.callarg/.svaID;

    • call sva_init_stack: to initialize the thread state in SVA; paramters:
      • kernel stack pointer: td2->td_kstack; // TODO:
      • stacklen;
      • new thread struct: td2;
      • kernel_thread_trampoline is passed in for proper start of a process (fork_exit()).
    • td2->callout = fork_return; //TODO:
    • td2->callarg = td2;
    • td2->sva = 1;
  • cpu_set_fork_handler(td,func,arg):

    • td->callout = func;
    • td->callarg = arg;
      code
  • cpu_exit(struct thread *td)

    • sva_release_stack(td->svaID)
  • cpu_set_syscall_retval(struct thread *td, int error)

    • use sva_icontext_setretval (td->td_retval[1], td->td_retval[0], 0); to replace the assignment to td->td_frame->tf_rax/rdx/rflags
    • use sva_icontext_restart(0,0).
  • cpu_set_upcall (struct thread *td, struct thread *td0)

    • use sva_init_stack() to initialize the new thread state in SVA;
  • add cpu_create_upcall(td,td0,func,arg)

sys/sys/amd64/amd64/machdep.c

  • exec_setregs(td, image_params *imgp, u_long stack) /* reset registers on default values on exec */

    • disable pcb fsbase/gsbase/clear_pcb_flags;
    • disable regs init;
    • add sva_translate(imgp->entry_addr);
    • add `sva_reinit_icontext(handle,0,((stack - 8) & ~-xFul) + 8, stack);
  • sys/vm/vm_glue.c

    • vm_thread_new(td,pages): init: td->svaID/mtx/sva/prev/callout/callarg;
  • sys/kern/kern_kthread.c

    • kthread_add(..) : use ‘cpu_create_upcall(newtd,oldtd,func,arg);’ to replace ‘cpu_set_upcall(newtd,oldtd); cpu_set_fork_handler(newtd,func,arg)’
  • sys/kern/kern_fork.c

    • fork_return(): set td->td_retval[0/1] = 0; call td->td_proc->p_sysent->sv_set_syscall_retval(td,0);
  • context switch: sys/kern/sched_ule|4bsd.c:

    • cpu_switch -> cpu_switch_sva
    • cpu_throw -> cpu_throw_sva

Signal

sys/amd64/amd64/sigtramp.S

  • NON_GPROF_ENTRY(sigcode): callq *%r8 (5th argument) instead of call *SIGF_HANDLER(%rsp) LLM: why this change?

sys/sys/amd64/amd64/machdep.c

  • sendsig (sig_t catcher, ksiginfo_t *ksi, sigset_t *mask):

    • disable assignment to sf.sf_uc.uc_mcontext.*;
    • disable assignment to regs-> rdi,rdx,rsi,rcx ;
    • disable copyout of sigfram to user stack;
    • disable reset of regs rsp,rip,rflags,cs,ds,es,fs,gs,flags,pcb_flags;
    • add sva_save_icontext()
    • add sva_ialloca(sigframe,..)
    • add sva_ipush_function5 ;
  • sys_sigreturn(td, struct sigreturn_args *uap):

    • disable load user context from ucp->uc_mcontext
    • diable set pcb flags;
    • add sva_load_icontext();

SVA memory management

sys/conf/ldscript.amd64

  • add .savmem section. _svastart, _svaend, SVAPTPages exported

sys/sys/amd64/amd64/

  • pmap.c:

    • create_pagetables(vm_paddr_t * firstaddr):
    • disable AMDID_PAGE1GB.
    • pmap_bootstrap(vm_paddr_t * firstaddr):
    • call sva_mmu_init(): set static address for kernel MMU initialization; init CR3;

      code

    • pmap_init_pat(): use sva_load_cr0 instead of load_cr0

      code

    • use sva_update_l2_mapping instead of pde_store(). same parameters.

      • in fuctions: pmap_update_pde_action(void *arg) / pmap_update_pde(…) / pmap_demote_pde(…) / pmap_protect_pde / pmap_promote_pde/ pmap_enter_pde / pmap_object_init_pt / pmap_pde_attr /
    • use sva_update_l1_mapping instead of pte_store. same parameters. in functions:

      • pmap_kenter(vm_offset_t va, vm_paddr_t pa) / pmap_kenter_attr(…) / pmap_qenter(…) / pmap_fill_ptp(…) / pmap_protect / pmap_enter_quick_locked / pmap_promote_pde / pmap_enter / pmap_change_wiring / pmap_remove_write / pmap_ts_referenced / pmap_clear_modify/ pmap_clear_reference / pmap_pte_attr /
    • use sva_remove_mapping(pte/ptq) instead of pte_clear(pte) pte_load_clear(ptq)

      • in fuctions: pmap_kremove(vm_offset_t va) / pmap_remove_pde / pmap_remove_pte / pmap_remove_all(vm_page_t m) / pmap_remove_pages /
    • add sva_remove_page in pmap_add_delayed_free_list.

    • _pmap_unwire_pte_hold(...): use sva_update_l2/l3/l4_mapping.

    • pmap_pinit(pmap_t pmap): update l4 mappings;

      • demote the PD Page and declare it to SVA: pmap_demote_DMAP(VM_PAGE_TO_PHYS(pml4pg)))
      • update l4 mappings sva_update_l4_mappings instead of `pmap->pm_pml4[KPML4I + i] = …;
    • pmap_allocpde -> _pmap_allocpte(…): update l4, l2, l3, l1 mappings.

    • pmap_release(): update l4;

    • pmap_growkernel(): l2/l3/l1

    • pmap_collect(): remove mapping

    • pmap_demote_pde(…): declare l1. update l2.

    • pmap_demote_pdpe: declare l2; up l2/l3.

    • pmap_promote_pde: update l2,l1

    • pmap_protect_pde: update l2

    • pmap_protect: update l1

    • pmap_enter: update l1,

    • pmap_enter_pde: up l2;

    • pmap_enter_quick_locked: up l1.

    • pmap_object_init_pt: up l2;

    • pmap_change_wiring: up l1.

    • pmap_copy: up l1/l2

    • pmap_remove_pages: rm mapping;

  • machdep.c: cpu_setregs():

    • disable load_cr0(cr0);
    • add sva_load_cr0(cr0).
  • amd64_mem.c: sva_load_cr0 ((cr0)) instead of load_cr0((cr0)) ==> LLM: cr0 must come from privileged sandbox. CR0 cannot be a global/shared variable.

  • sys/kern/kern_sva.c

    • provideSVAMemory(size): OS allocate memory and pass it to SVA to use.
    • releaseSVAMemory(paddr,size): SVA call this to release memory back to OS
    • testSVAMemory(unsigned char *p): try to access secure memory from OS: should fail.

IO

sys/amd64/amd64/sys_machdep.c

  • “SVA: no support for I/O permissions!”

FPU

sys/sys/amd64/amd64/

  • fpu.c
    • empty fpuexit(struct thread *td)
    • empty fpudna(void)
    • empty fpugetregs(struct thread *td)
    • empty fpusetregs(struct thread *td, struct savefpu *addr)

Support funcs

sys/amd64/amd64/

  • support.S
    • copyin/out changed to real_copyin/out
    • casuword/32 -> real_casuword/32
    • fuword/32/16 -> real_fuword64/32/16
    • fubyte -> real_fubyte
    • suword/32/16 -> real_suword64/32/16
    • RETQ instead of ret;
  • sva_support.c
    • copyinstr(…): sva_invokestrncpy(…)
    • copyin/out(…): sva_invoke(…,real_copyin/out)
    • fuword64/32/16/fuword: sva_invoke(…,real_fuword64/32/16)
    • fubyte/subyte/suword/16/32/64, casuword/32 -> svainvoke*

Configurations

sys/conf/

  • files: add kern/kern_sva.c
  • files.amd64: add amd64/amd64/sva_support.c
  • kern.mk: turn on CFI and SFI for clang and SVA: CFLAGS+= -mllvm -add-sfi -mllvm -enable-sfi-loadchecks -Xclang -backend-option -Xclang -x86-add-cfi
  • kern.post.mk: compile libsva.a into full kernel binary:

    code

  • ldscript.amd64: add ‘svamem’ section:

    code

  • NOTES: add makeoptions CONF_CFLAGS=-fno-builtin -I/usr/home/criswell/src/sva/SVA/include

  • options: add SVA_MMU opt_sva_mmu.h

sys/dev/aic7xxx/aicasm/Makefile: add CC=gcc

TODONES

TODO: []

DONE:

sys/dev/aic7xxx/aicasm/Makefile

sys/conf/

sys/amd64/amd64/

  • vm_machdep.c
  • trap.c
  • sigtramp.S
  • support.S
  • sva_support.c
  • sys_machdep.c
  • pmap.c
  • mp_machdep.c
  • machdep.c
  • fpu.c
  • apic_vector.S; atpic_vector.S; cpu_switch.S; exception.S; locore.S;
  • amd64_mem.c

sys/amd64

-/conf/{SVA, SVAMAC}; -/ia32/ia32_syscall.c; -/include/ {apicvar.h, asm.h, asmacros.h, cpufunc.h, pmap.h}; -/linux32/linux32_locore.S;

sys/kern/; sys/sys/; sys/vm/; sys/x86/x86/local_apic.c;

Reference 1


  1. SVA: github.com/jtcriswell/SVA.git; or tupipa/sva_freebsd90 ↩
Created Oct 6, 2019 // Last Updated Oct 14, 2019

If you could revise
the fundmental principles of
computer system design
to improve security...

... what would you change?