Today I delve deeper into the bug that's keep occuring. I added bunch of debug messages throughout the kernel to see what parts of code is not working right.

1. Binary Handlers

To dump the addresses of elf format handlers to identify what handler has what memory address, I added these codes :

/* at binfmt_my_elf.c */ 
static int __init init_elf_binfmt(void)
{
	register_binfmt(&elf_format);
	printk("myelf : elf_format : 0x%lx\n" , (unsigned long)&elf_format);
    ...

/* at binfmt_elf.c */ 
static int __init init_elf_binfmt(void)
{
	register_binfmt(&elf_format);
	printk("real elf : elf_format : 0x%lx\n" , (unsigned long)&elf_format);
    ...

For whatever reason the elf binary handler is registered twice to the kernel, so I just made a code that ignores the registration of system call table when the identical system call is already registered to the system.

2. System Call Tables

To dump the addresses of system call tables of the current task and bprm structure, I added these codes :

/* In exec.c, at search_binary_handler() */
...
if (bprm->point_of_no_return || (retval != -ENOEXEC)) {
    // success
    // set system call handler
    bprm->sys_call_tbl_ptr = sys_tbl_search_by_bin_handler(fmt);
    
    // if binary format is not registered, just use default table
    if(bprm->sys_call_tbl_ptr == 0x00) {
        printk("binary handler not registered to dynamic syscall list! using default table\n");
        bprm->sys_call_tbl_ptr = sys_tbl_get_default_table();
    }
    else {
        printk("using registered dynamic syscall, table : 0x%lx , fmt : 0x%lx\n" , (unsigned long)bprm->sys_call_tbl_ptr , (unsigned long)fmt);
    }
    printk("bprm->sys_call_tbl_ptr : 0x%lx\n" , (unsigned long)bprm->sys_call_tbl_ptr);
    current->sys_call_tbl = (unsigned long)bprm->sys_call_tbl_ptr;
    printk("current->sys_call_tbl : 0x%lx\n" , current->sys_call_tbl);

    read_unlock(&binfmt_lock);
    return retval;
}
...

As you can see the system call table is properly assigned by the binary handler.

0xffffffffbaa0d160 is the address of the standard linux system call table. 0xffffffffbaa0d130 is the address of the modified system call table.

3. Global System Call Tables

To dump the global system call table, I created a system call that prints out the pointer of global system call table.

548  common    sys_get_current_syscall_tbl   __sys_get_current_syscall_tbl

The system call handler :

asmlinkage long __x64___sys_get_current_syscall_tbl(void) {
    printk("(syscall) current system call table : 0x%lx\n" , (unsigned long)sys_call_table);
    return 0x00;
}

Thus, when program calls #548 system call, you would get the address of current global system call table. I made a test program that calls the #548 system call.

#include <unistd.h>
#include <sys/syscall.h>

int main(void) {
    syscall(548);
    return 0;
}

Upon calling the program, we can see that there is a discrepency between the system call table stored in the task_struct and the global system call table. You can see that the system call table stored in the task_struct has the expected table but the global system call table is not matching with the task_struct's table.

4. Reason of Problem

The global system call table not switching properly means that there is an error or some unknown factor (that I've not figured yet) in the context switching of the system call table. When we keep execute the program we can observe that the global system call table is changing, which tells us that the context switching happening here is very unstable and unreliable.

It's keep changing!!

Although system call table keep changing is a good thing, as we know that the context switching itself is properly working, the context switching must be done to set the syscall table to corresponding syscall table of current task.

We need to find what is causing the error or just find a new method to achieve this goal..