linux-insides/Theory/linux-theory-3.md

Inline assembly

Introduction

While reading source code in the Linux kernel, I often see statements like this:

__asm__("andq %%rsp,%0; ":"=r" (ti) : "0" (CURRENT_MASK));

Yes, this is inline assembly or in other words assembler code which is integrated in a high level programming language. In this case the high level programming language is C. Yes, the C programming language is not very high-level, but still.

If you are familiar with the assembly programming language, you may notice that inline assembly is not very different from normal assembler. Moreover, the special form of inline assembly which is called basic form is exactly the same. For example:

__asm__("movq %rax, %rsp");

or:

__asm__("hlt");

The same code (of course without __asm__ prefix) you might see in plain assembly code. Yes, this is very similar, but not so simple as it might seem at first glance. Actually, the GCC supports two forms of inline assembly statements:

• basic;
• extended.

The basic form consists of only two things: the __asm__ keyword and the string with valid assembler instructions. For example it may look something like this:

__asm__("movq    $3, %rax\t\n"
        "movq    %rsi, %rdi");

The asm keyword may be used in place of __asm__, however __asm__ is portable whereas the asm keyword is a GNU extension. In further examples I will only use the __asm__ variant.

If you know assembly programming language this looks pretty familiar. The main problem is in the second form of inline assembly statements - extended. This form allows us to pass parameters to an assembly statement, perform jumps etc. Does not sound difficult, but requires knowledge of special rules in addition to knowledge of the assembly language. Every time I see yet another piece of inline assembly code in the Linux kernel, I need to refer to the official documentation of GCC to remember how a particular qualifier behaves or what the meaning of =&r is for example.

I've decided to write this part to consolidate my knowledge related to the inline assembly, as inline assembly statements are quite common in the Linux kernel and we may see them in linux-insides parts sometimes. I thought that it would be useful if we have a special part which contains information on more important aspects of the inline assembly. Of course you may find comprehensive information about inline assembly in the official documentation, but I like to put everything in one place.

** Note: This part will not provide guide for assembly programming. It is not intended to teach you to write programs with assembler or to know what one or another assembler instruction means. Just a little memo for extended asm. **

Introduction to extended inline assembly

So, let's start. As I already mentioned above, the basic assembly statement consists of the asm or __asm__ keyword and set of assembly instructions. This form is in no way different from "normal" assembly. The most interesting part is inline assembler with operands, or extended assembler. An extended assembly statement looks more complicated and consists of more than two parts:

__asm__ [volatile] [goto] (AssemblerTemplate
                           [ : OutputOperands ]
                           [ : InputOperands  ]
                           [ : Clobbers       ]
                           [ : GotoLabels     ]);

All parameters which are marked with squared brackets are optional. You may notice that if we skip the optional parameters and the modifiers volatile and goto we obtain the basic form.

Let's start to consider this in order. The first optional qualifier is volatile. This specifier tells the compiler that an assembly statement may produce side effects. In this case we need to prevent compiler optimizations related to the given assembly statement. In simple terms the volatile specifier instructs the compiler not to modify the statement and place it exactly where it was in the original code. As an example let's look at the following function from the Linux kernel:

static inline void native_load_gdt(const struct desc_ptr *dtr)
{
	asm volatile("lgdt %0"::"m" (*dtr));
}

Here we see the native_load_gdt function which loads a base address from the Global Descriptor Table to the GDTR register with the lgdt instruction. This assembly statement is marked with volatile qualifier. It is very important that the compiler does not change the original place of this assembly statement in the resulting code. Otherwise the GDTR register may contain wrong address for the Global Descriptor Table or the address may be correct, but the structure has not been filled yet. This can lead to an exception being generated, preventing the kernel from booting correctly.

The second optional qualifier is the goto. This qualifier tells the compiler that the given assembly statement may perform a jump to one of the labels which are listed in the GotoLabels. For example:

__asm__ goto("jmp %l[label]" : : : : label);

Since we finished with these two qualifiers, let's look at the main part of an assembly statement body. As we have seen above, the main part of an assembly statement consists of the following four parts:

• set of assembly instructions;
• output parameters;
• input parameters;
• clobbers.

The first represents a string which contains a set of valid assembly instructions which may be separated by the \t\n sequence. Names of processor registers must be prefixed with the %% sequence in extended form and other symbols like immediates must start with the $ symbol. The OutputOperands and InputOperands are comma-separated lists of C variables which may be provided with "constraints" and the Clobbers is a list of registers or other values which are modified by the assembler instructions from the AssemblerTemplate beyond those listed in the OutputOperands. Before we dive into the examples we have to know a little bit about constraints. A constraint is a string which specifies placement of an operand. For example the value of an operand may be written to a processor register or read from memory etc.

Consider the following simple example:

#include <stdio.h>

int main(void)
{
        unsigned long a = 5;
        unsigned long b = 10;
        unsigned long sum = 0;

        __asm__("addq %1,%2" : "=r" (sum) : "r" (a), "0" (b));
        printf("a + b = %lu\n", sum);
        return 0;
}

Let's compile and run it to be sure that it works as expected:

$ gcc test.c -o test
./test
a + b = 15

Ok, great. It works. Now let's look at this example in detail. Here we see a simple C program which calculates the sum of two variables placing the result into the sum variable and in the end we print the result. This example consists of three parts. The first is the assembly statement with the add instruction. It adds the value of the source operand together with the value of the destination operand and stores the result in the destination operand. In our case:

addq %1, %2

will be expanded to the:

addq a, b

Variables and expressions which are listed in the OutputOperands and InputOperands may be matched in the AssemblerTemplate. An input/output operand is designated as %N where the N is the number of operand from left to right beginning from zero. The second part of the our assembly statement is located after the first : symbol and contains the definition of the output value:

"=r" (sum)

Notice that the sum is marked with two special symbols: =r. This is the first constraint that we have encountered. The actual constraint here is only r itself. The = symbol is modifier which denotes output value. This tells to compiler that the previous value will be discarded and replaced by the new data. Besides the = modifier, GCC provides support for following three modifiers:

• + - an operand is read and written by an instruction;
• & - output register shouldn't overlap an input register and should be used only for output;
• % - tells the compiler that operands may be commutative.

Now let's go back to the r qualifier. As I mentioned above, a qualifier denotes the placement of an operand. The r symbol means a value will be stored in one of the general purpose register. The last part of our assembly statement:

"r" (a), "0" (b)

These are input operands - variables a and b. We already know what the r qualifier does. Now we can have a look at the constraint for the variable b. The 0 or any other digit from 1 to 9 is called "matching constraint". With this a single operand can be used for multiple roles. The value of the constraint is the source operand index. In our case 0 will match sum. If we look at assembly output of our program:

0000000000400400 <main>:
  ...
  ...
  ...
  4004fe:       48 c7 45 f8 05 00 00    movq   $0x5,-0x8(%rbp)
  400506:       48 c7 45 f0 0a 00 00    movq   $0xa,-0x10(%rbp)

  400516:       48 8b 55 f8             mov    -0x8(%rbp),%rdx
  40051a:       48 8b 45 f0             mov    -0x10(%rbp),%rax
  40051e:       48 01 d0                add    %rdx,%rax

First of all our values 5 and 10 will be put at the stack and then these values will be moved to the two general purpose registers: %rdx and %rax.

This way the %rax register is used for storing the value of the b as well as storing the result of the calculation. NOTE that I've used gcc 6.3.1 version, so the resulted code of your compiler may differ.

We have looked at input and output parameters of an inline assembly statement. Before we move on to other constraints supported by gcc, there is one remaining part of the inline assembly statement we have not discussed yet - clobbers.

Clobbers

As mentioned above, the "clobbered" part should contain a comma-separated list of registers whose content will be modified by the assembler code. This is useful if our assembly expression needs additional registers for calculation. If we add clobbered registers to the inline assembly statement, the compiler take this into account and the register in question will not simultaneously be used by the compiler.

Consider the example from before, but we will add an additional, simple assembler instruction:

__asm__("movq $100, %%rdx\t\n"
        "addq %1,%2" : "=r" (sum) : "r" (a), "0" (b));

If we look at the assembly output:

0000000000400400 <main>:
  ...
  ...
  ...
  4004fe:       48 c7 45 f8 05 00 00    movq   $0x5,-0x8(%rbp)
  400506:       48 c7 45 f0 0a 00 00    movq   $0xa,-0x10(%rbp)

  400516:       48 8b 55 f8             mov    -0x8(%rbp),%rdx
  40051a:       48 8b 45 f0             mov    -0x10(%rbp),%rax

  40051e:       48 c7 c2 64 00 00 00    mov    $0x64,%rdx
  400525:       48 01 d0                add    %rdx,%rax

we will see that the %rdx register is overwritten with 0x64 or 100 and the result will be 110 instead of 10. Now if we add the %rdx register to the list of clobbered registers:

__asm__("movq $100, %%rdx\t\n"
        "addq %1,%2" : "=r" (sum) : "r" (a), "0" (b) : "%rdx");

and look at the assembler output again:

0000000000400400 <main>:
  4004fe:       48 c7 45 f8 05 00 00    movq   $0x5,-0x8(%rbp)
  400506:       48 c7 45 f0 0a 00 00    movq   $0xa,-0x10(%rbp)

  400516:       48 8b 4d f8             mov    -0x8(%rbp),%rcx
  40051a:       48 8b 45 f0             mov    -0x10(%rbp),%rax

  40051e:       48 c7 c2 64 00 00 00    mov    $0x64,%rdx
  400525:       48 01 c8                add    %rcx,%rax

the %rcx register will be used for sum calculation, preserving the intended semantics of the program. Besides general purpose registers, we may pass two special specifiers. They are:

• cc;
• memory.

The first - cc indicates that an assembler code modifies flags register. This is typically used if the assembly within contains arithmetic or logic instructions:

__asm__("incq %0" ::""(variable): "cc");

The second memory specifier tells the compiler that the given inline assembly statement executes read/write operations on memory not specified by operands in the output list. This prevents the compiler from keeping memory values loaded and cached in registers. Let's take a look at the following example:

#include <stdio.h>

int main(void)
{
        unsigned long a[3] = {10000000000, 0, 1};
        unsigned long b = 5;
        
        __asm__ volatile("incq %0" :: "m" (a[0]));

        printf("a[0] - b = %lu\n", a[0] - b);
        return 0;
}

This example may be artificial, but it illustrates the main idea. Here we have an array of integers and one integer variable. The example is pretty simple, we take the first element of a and increment its value. After this we subtract the value of b from the first element of a. In the end we print the result. If we compile and run this simple example the result may surprise you:

~$ gcc -O3  test.c -o test
~$ ./test
a[0] - b = 9999999995

The result is a[0] - b = 9999999995 here, but why? We incremented a[0] and subtracted b, so the result should be a[0] - b = 9999999996 here.

If we have a look at the assembler output for this example:

00000000004004f6 <main>:
  4004b4:       48 b8 00 e4 0b 54 02    movabs $0x2540be400,%rax
  4004be:       48 89 04 24             mov    %rax,(%rsp)
  ...
  ...
  ...
  40050e:       ff 44 24 f0             incq   (%rsp)

  4004d8:       48 be fb e3 0b 54 02    movabs $0x2540be3fb,%rsi

we will see that the first element of the a contains the value 0x2540be400 (10000000000). The last two lines of code are the actual calculations.

We see our increment instruction with incq but then just a move of 0x2540be3fb (9999999995) to the %rsi register. This looks strange.

The problem is we have passed the -O3 flag to gcc, so the compiler did some constant folding and propagation to determine the result of a[0] - 5 at compile time and reduced it to a movabs with a constant 0x2540be3fb or 9999999995 in runtime.

Let's now add memory to the clobbers list:

__asm__ volatile("incq %0" :: "m" (a[0]) : "memory");

and the new result of running this is:

~$ gcc -O3  test.c -o test
~$ ./test
a[0] - b = 9999999996

Now the result is correct. If we look at the assembly output again:

00000000004004f6 <main>:
  400404:       48 b8 00 e4 0b 54 02    movabs $0x2540be400,%rax
  40040b:       00 00 00 
  40040e:       48 89 04 24             mov    %rax,(%rsp)
  400412:       48 c7 44 24 08 00 00    movq   $0x0,0x8(%rsp)
  400419:       00 00 
  40041b:       48 c7 44 24 10 01 00    movq   $0x1,0x10(%rsp)
  400422:       00 00 
  400424:       48 ff 04 24             incq   (%rsp)
  400428:       48 8b 04 24             mov    (%rsp),%rax
  400431:       48 8d 70 fb             lea    -0x5(%rax),%rsi

we will see one difference here which is in the last two lines:

  400428:       48 8b 04 24             mov    (%rsp),%rax
  400431:       48 8d 70 fb             lea    -0x5(%rax),%rsi

Instead of constant folding, GCC now preserves calculations in the assembly and places the value of a[0] in the %rax register afterwards. In the end it just subtracts the constant value of b from the %rax register and puts result to the %rsi.

Besides the memory specifier, we also see a new constraint here - m. This constraint tells the compiler to use the address of a[0], instead of its value. So, now we are finished with clobbers and we may continue by looking at other constraints supported by GCC besides r and m which we have already seen.

Constraints

Now that we are finished with all three parts of an inline assembly statement, let's return to constraints. We already saw some constraints in the previous parts, like r which represents a register operand, m which represents a memory operand and 0-9 which represent an reused, indexed operand. Besides these GCC provides support for other constraints. For example the i constraint represents an immediate integer operand with know value:

#include <stdio.h>

int main(void)
{
        int a = 0;

        __asm__("movl %1, %0" : "=r"(a) : "i"(100));
        printf("a = %d\n", a);
        return 0;
}

The result is:

~$ gcc test.c -o test
~$ ./test
a = 100

Or for example I which represents an immediate 32-bit integer. The difference between i and I is that i is general, whereas I is strictly specified to 32-bit integer data. For example if you try to compile the following code:

unsigned long test_asm(int nr)
{
        unsigned long a = 0;

        __asm__("movq %1, %0" : "=r"(a) : "I"(0xffffffffffff));
        return a;
}

you will get an error:

$ gcc -O3 test.c -o test
test.c: In function ‘test_asm’:
test.c:7:9: warning: asm operand 1 probably doesn’t match constraints
         __asm__("movq %1, %0" : "=r"(a) : "I"(0xffffffffffff));
         ^
test.c:7:9: error: impossible constraint in ‘asm’

when at the same time:

unsigned long test_asm(int nr)
{
        unsigned long a = 0;

        __asm__("movq %1, %0" : "=r"(a) : "i"(0xffffffffffff));
        return a;
}

works perfectly:

~$ gcc -O3 test.c -o test
~$ echo $?
0

GCC also supports J, K, N constraints for integer constants in the range of 0-63 bits, signed 8-bit integer constants and unsigned 8-bit integer constants respectively. The o constraint represents a memory operand with an offsetable memory address. For example:

#include <stdio.h>

int main(void)
{
        static unsigned long arr[3] = {0, 1, 2};
        static unsigned long element;
        
        __asm__ volatile("movq 16+%1, %0" : "=r"(element) : "o"(arr));
        printf("%lu\n", element);
        return 0;
}

The result, as expected:

~$ gcc -O3 test.c -o test
~$ ./test
2

All of these constraints may be combined (so long as they do not conflict). In this case the compiler will choose the best one for a certain situation. For example:

unsigned long a = 10;
unsigned long b = 20;

void main(void)
{
    __asm__ ("movq %1,%0" : "=mr"(b) : "rm"(a));
}

will use a memory operand:

main:
        movq a(%rip),b(%rip)
        ret
b:
        .quad   20
a:
        .quad   10

instead of direct usage of general purpose registers.

That's about all of the commonly used constraints in inline assembly statements. You can find more in the official documentation.

Architecture specific constraints

Before we finish, let's look at the set of special constraints. These constrains are architecture specific and as this book is specific to the x86_64 architecture, we will look at constraints related to it. First of all the set of a ... d and also S and D constraints represent generic purpose registers. In this case the a constraint corresponds to %al, %ax, %eax or %rax register depending on instruction size. The S and D constraints are %si and %di registers respectively. For example let's take our previous example. We can see in its assembly output that value of the a variable is stored in the %eax register. Now let's look at the assembly output of the same assembly, but with other constraint:

#include <stdio.h>

int a = 1;

int main(void)
{
        int b;
        __asm__ ("movq %1,%0" : "=r"(b) : "d"(a));
        return b;
}

Now we see that value of the a variable will be stored in the %rax register:

0000000000400400 <main>:
  4004aa:       48 8b 05 6f 0b 20 00    mov    0x200b6f(%rip),%rax        # 601020 <a>

The f and t constraints represent any floating point stack register - %st and the top of the floating point stack respectively. The u constraint represents the second value from the top of the floating point stack.

That's all. You may find more details about x86_64 and general constraints in the official documentation.

Links

• Linux kernel source code
• assembly programming language
• GCC
• GNU extension
• Global Descriptor Table
• Processor registers
• add instruction
• flags register
• x86_64
• constraints