Embedded Systems.. EASY questions!

embedded_linux_realtime.pdf

Home >Computer Science homework help >Embedded Systems.. EASY questions!

1 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Embedded Linux system development

Realtime in embedded Linux

systems Michael Opdenacker Thomas Petazzoni

Gilles Chanteperdrix Free Electrons

© Copyright 20042011, Free Electrons. Creative Commons BYSA 3.0 license Latest update: Feb 21, 2011, Document sources, updates and translations: http://freeelectrons.com/docs/realtime Corrections, suggestions, contributions and translations are welcome!

http://free-electrons.com/docs/realtime

2 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Real Time in Embedded Linux Systems

Introduction

3 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Embedded Linux and real time

Due to its advantages, Linux and the opensource softwares are more and more commonly used in embedded applications

However, some applications also have realtime constraints

They, at the same time, want to

Get all the nice advantages of Linux: hardware support, components reuse, low cost, etc.

Get their realtime constraints met

4 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Embedded Linux and real time

Linux is an operating system part of the large Unix family

It was originally designed as a timesharing system

The main goal is to get the best throughput from the available hardware, by making the best possible usage of resources (CPU, memory, I/O)

Time determinism is not taken into account

On the opposite, realtime constraints imply time determinism, even at the expense of lower global throughput

Best throughput and time determinism are contradictory requirements

5 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Linux and realtime approaches

Over time, two major approaches have been taken to bring real time requirements into Linux

Approach 1

Improve the Linux kernel itself so that it matches realtime requirements, by providing bounded latencies, realtime APIs, etc.

Approach taken by the mainline Linux kernel and the PREEMPT_RT project.

Approach 2

Add a layer below the Linux kernel that will handle all the realtime requirements, so that the behaviour of Linux doesn't affect realtime tasks.

Approach taken by RTLinux, RTAI and Xenomai

6 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Approach 1 Improving the main Linux kernel with

PREEMPT_RT

7 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Understanding latency

When developing realtime applications with a system such as Linux, the typical scenario is the following

An event from the physical world happens and gets notified to the CPU by means of an interrupt

The interrupt handler recognizes and handles the event, and then wakeup the userspace task that will react to this event

Some time later, the userspace task will run and be able to react to the physical world event

Realtime is about providing guaranteed worst case latencies for this reaction time, called latency

Something not very important... Your important realtime task !

Interrupt ! ?

8 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Linux kernel latency components

Waiting task

interrupt latency

Interrupt handler Scheduler

Running task

Interrupt

Scheduling latency

scheduler latency

scheduler duration

Process context

Interrupt context

Makes the task runnable

kernel latency = interrupt latency + handler duration + scheduler latency + scheduler duration

handler duration

9 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Interrupt latency

Waiting task

interrupt latency

Interrupt handler Scheduler

Running task

Interrupt

handler duration

scheduler latency

scheduler duration

Makes the task runnable

Time elapsed before executing the interrupt handler

Scheduling latency

10 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Source of interrupt latency

One of the concurrency prevention mechanism used in the kernel is the spinlock

It has several variants, but one of the variant commonly used to prevent concurrent accesses between a process context and an interrupt context works by disabling interrupts

Critical sections protected by spinlocks, or other section in which interrupts are explictly disabled will delay the beginning of the execution of the interrupt handler

The duration of these critical sections is unbounded

Other possible source: shared interrupts

Kernel code

Critical section protected by spinlock

Interrupt handler

Interrupt ?

11 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Interrupt handler duration

Waiting task

interrupt latency

Interrupt handler Scheduler

Running task

Interrupt

handler duration

scheduler latency

scheduler duration

Makes the task runnable

Time taken to execute the interrupt handler

Scheduling latency

12 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Interrupt handler implementation

In Linux, many interrupt handlers are split in two parts

A tophalf, started by the CPU as soon as interrupt are enabled. It runs with the interrupt line disabled and is supposed to complete as quickly as possible.

A bottomhalf, scheduled by the tophalf, which starts after all pending tophalf have completed their execution.

Therefore, for realtime critical interrupts, bottomhalf shouldn't be used: their execution is delayed by all other interrupts in the system.

Top half

Interrupt ACK Exit

Bottom half

Schedule bottom half

Other interrupt handlers...

Handle device data...

Wake up waiting tasks

User space...

13 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Scheduler latency

Waiting task

interrupt latency

Interrupt handler Scheduler

Running task

Interrupt

handler duration

scheduler latency

scheduler duration

Makes the task runnable

Time elapsed before executing the scheduler

Scheduling latency

14 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Understanding preemption (1)

The Linux kernel is a preemptive operating system

When a task runs in userspace mode and gets interrupted by an interruption, if the interrupt handler wakes up another task, this task can be scheduled as soon as we return from the interrupt handler.

Task A (running in user mode)

Interrupt handler Wakes up Task B

Task B (running in user mode)

Interrupt

15 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Understanding preemption (2)

However, when the interrupt comes while the task is executing a system call, this system call has to finish before another task can be scheduled.

By default, the Linux kernel does not do kernel preemption.

This means that the time before which the scheduler will be called to schedule another task is unbounded.

Task A (user mode)

Interrupt handler Wakes up Task B

Task B (user mode)

System call

Task A (kernel mode)

Interrupt

? Return from syscall

16 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Scheduler duration

Waiting task

interrupt latency

Interrupt handler Scheduler

Running task

Interrupt

handler duration

scheduler latency

scheduler duration

Makes the task runnable

Time taken to execute the scheduler and switch to the new task.

Scheduling latency

17 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Other nondeterministic mechanisms

Outside of the critical path detailed previously, other non deterministic mechanisms of Linux can affect the execution time of realtime tasks

Linux is highly based on virtual memory, as provided by an MMU, so that memory is allocated on demand. Whenever an application accesses code or data for the first time, it is loaded on demand, which can creates huge delays.

Many C library services or kernel services are not designed with realtime constraints in mind.

18 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Priority inversion

Acquires a lock

Priority

Time

preempted

Tries to get the same

lock waits

A process with a low priority might hold a lock needed by a higher priority process, effectively reducing the priority of this process. Things can be even worse if a middle priority process uses the CPU.

19 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Interrupt handler priority

top priority task

Any interrupt

top priority task

Any interrupt...

In Linux, interrupt handlers are executed directly by the CPU interrupt mechanisms, and not under control of the Linux scheduler. Therefore, all interrupt handlers have an higher priority than all tasks running on the system.

20 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

The PREEMPT_RT project

Longterm project lead by Linux kernel developers Ingo Molnar, Thomas Gleixner and Steven Rostedt

https://rt.wiki.kernel.org

The goal is to gradually improve the Linux kernel regarding real time requirements and to get these improvements merged into the mainline kernel

PREEMPT_RT development works very closely with the mainline development

Many of the improvements designed, developed and debugged inside PREEMPT_RT over the years are now part of the mainline Linux kernel

The project is a longterm branch of the Linux kernel that ultimately should disappear as everything will have been merged

https://rt.wiki.kernel.org/

21 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Improvements in the mainline kernel

Coming from the PREEMPT_RT project

Since the beginning of 2.6

O(1) scheduler

Kernel preemption

Better POSIX realtime API support

Since 2.6.18

Priority inheritance support for mutexes

Since 2.6.21

Highresolution timers

Since 2.6.30

Threaded interrupts

Since 2.6.33

Spinlock annotations

22 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

New preemption options in Linux 2.6

2 new preemption models offered by standard Linux 2.6:

23 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

1st option: no forced preemption

CONFIG_PREEMPT_NONE Kernel code (interrupts, exceptions, system calls) never preempted. Default behavior in standard kernels.

Best for systems making intense computations, on which overall throughput is key.

Best to reduce task switching to maximize CPU and cache usage (by reducing context switching).

Still benefits from some Linux 2.6 improvements: O(1) scheduler, increased multiprocessor safety (work on RT preemption was useful to identify hard to find SMP bugs).

Can also benefit from a lower timer frequency (100 Hz instead of 250 or 1000).

24 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

2nd option: voluntary kernel preemption

CONFIG_PREEMPT_VOLUNTARY Kernel code can preempt itself

Typically for desktop systems, for quicker application reaction to user input.

Adds explicit rescheduling points throughout kernel code.

Minor impact on throughput.

25 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

3rd option: preemptible kernel

CONFIG_PREEMPT Most kernel code can be involuntarily preempted at any time. When a process becomes runnable, no more need to wait for kernel code (typically a system call) to return before running the scheduler.

Exception: kernel critical sections (holding spinlocks), but a rescheduling point occurs when exiting the outer critical section, in case a preemption opportunity would have been signaled while in the critical section.

Typically for desktop or embedded systems with latency requirements in the milliseconds range.

Still a relatively minor impact on throughput.

26 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Priority inheritance

One classical solution to the priority inversion problem is called priority inheritance

The idea is that when a task of a low priority holds a lock requested by an higher priority task, the priority of the first task gets temporarly raised to the priority of the second task : it has inherited its priority.

In Linux, since 2.6.18, mutexes support priority inheritance

In userspace, priority inheritance must be explictly enabled on a permutex basis.

27 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

High resolution timers

The resolution of the timers used to be bound to the resolution of the regular system tick

Usually 100 Hz or 250 Hz, depending on the architecture and the configuration

A resolution of only 10 ms or 4 ms.

Increasing the regular system tick frequency is not an option as it would consume too much resources

The highresolution timers infrastructure, merged in 2.6.21, allows to use the available hardware timers to program interrupts at the right moment.

Hardware timers are multiplexed, so that a single hardware timer is sufficient to handle a large number of softwareprogrammed timers.

Usable directly from userspace using the usual timer APIs

28 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Threaded interrupts

To solve the interrupt inversion problem, PREEMPT_RT has introduced the concept of threaded interrupts

The interrupt handlers run in normal kernel threads, so that the priorities of the different interrupt handlers can be configured

The real interrupt handler, as executed by the CPU, is only in charge of masking the interrupt and wakingup the corresponding thread

The idea of threaded interrupts also allows to use sleeping spinlocks (see later)

Merged since 2.6.30, the conversion of interrupt handlers to threaded interrupts is not automatic : drivers must be modified

In PREEMPT_RT, all interrupt handlers are switched to threaded interrupts

29 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

PREEMPT_RT specifics

30 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

CONFIG_PREEMPT_RT (1)

The PREEMPT_RT patch adds a new « level » of preemption, called CONFIG_PREEMPT_RT

This level of preemption replaces all kernel spinlocks by mutexes (or socalled sleeping spinlocks)

Instead of providing mutual exclusion by disabling interrupts and preemption, they are just normal locks : when contention happens, the process is blocked and another one is selected by the scheduler

Works well with threaded interrupts, since threads can block, while usual interrupt handlers could not

Some core, carefully controlled, kernel spinlocks remain as normal spinlocks

31 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

CONFIG_PREEMPT_RT (2)

With CONFIG_PREEMPT_RT, virtually all kernel code becomes preemptible

An interrupt can occur at any time, when returning from the interrupt handler, the woken up process can start immediately

This is the last big part of PREEMPT_RT that isn't fully in the mainline kernel yet

Part of it has been merged in 2.6.33 : the spinlock annotations. The spinlocks that must remain as spinning spinlocks are now differentiated from spinlocks that can be converted to sleeping spinlocks. This has reduced a lot the PREEMPT_RT patch size !

32 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Threaded interrupts

The mechanism of threaded interrupts in PREEMPT_RT is still different from the one merged in mainline

In PREEMPT_RT, all interrupt handlers are unconditionally converted to threaded interrupts.

This is a temporary solution, until interesting drivers in mainline get gradually converted to the new threaded interrupt API that has been merged in 2.6.30.

33 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Setting up PREEMPT_RT

34 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

PREEMPT_RT setup (1)

PREEMPT_RT is delivered as a patch against the mainline kernel

Best to have a board supported by the mainline kernel, otherwise the PREEMPT_RT patch may not apply and may require some adaptations

Many official kernel releases are supported, but not all. For example, 2.6.31 and 2.6.33 are supported, but not 2.6.32.

Quick set up

Download and extract mainline kernel

Download the corresponding PREEMPT_RT patch

Apply it to the mainline kernel tree

35 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

PREEMPT_RT setup (2)

In the kernel configuration, be sure to enable

CONFIG_PREEMPT_RT

Highresolution timers

Compile your kernel, and boot

You are now running the realtime Linux kernel

Of course, some system configuration remains to be done, in particular setting appropriate priorities to the interrupt threads, which depend on your application.

36 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Realtime application development

37 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Development and compilation

No special library is needed, the POSIX realtime API is part of the standard C library

The glibc or eglibc C libraries are recommended, as the support of some realtime features is not available yet in uClibc

Priority inheritance mutexes or NPTL on some architectures, for example

Compile a program

ARCHlinuxgcc o myprog myprog.c lrt

To get the documentation of the POSIX API

Install the manpagesposixdev package

Run man functioname

38 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Process, thread ?

Confusion about the terms «process», «thread» and «task»

In Unix, a process is created using fork() and is composed of

An address space, which contains the program code, data, stack, shared libraries, etc.

One thread, that starts executing the main() function.

Upon creation, a process contains one thread

Additional threads can be created inside an existing process, using pthread_create()

They run in the same address space as the initial thread of the process

They start executing a function passed as argument to pthread_create()

39 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Process, thread: kernel point of view

The kernel represents each thread running in the system by a structure of type task_struct

From a scheduling point of view, it makes no difference between the initial thread of a process and all additional threads created dynamically using pthread_create()

Address space

Thread A

Process after fork()

Address space

Thread A

Thread B

Same process after pthread_create()

40 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Creating threads

Linux support the POSIX thread API

To create a new thread

pthread_create(pthread_t *thread, pthread_attr_t *attr, void *(*routine)(*void*), void *arg);

The new thread will run in the same address space, but will be scheduled independently

Exiting from a thread

pthread_exit(void *value_ptr);

Waiting for a thread termination

pthread_join(pthread_t *thread, void **value_ptr);

41 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Scheduling classes (1)

The Linux kernel scheduler support different scheduling classes

The default class, in which processes are started by default is a timesharing class

All processes, regardless of their priority, get some CPU time

The proportion of CPU time they get is dynamic and affected by the nice value, which ranges from 20 (highest) to 19 (lowest). Can be set using the nice or renice commands

The realtime classes SCHED_FIFO and SCHED_RR

The highest priority process gets all the CPU time, until it blocks.

In SCHED_RR, roundrobin scheduling between the processes of the same priority. All must block before lower priority processes get CPU time.

Priorities ranging from 0 (lowest) to 99 (highest)

42 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Scheduling classes (2)

An existing program can be started in a specific scheduling class with a specific priority using the chrt command line tool

Example: chrt f 99 ./myprog

The sched_setscheduler() API can be used to change the scheduling class and priority of a process

int sched_setscheduler(pid_t pid, int policy, const struct sched_param *param);

policy can be SCHED_OTHER, SCHED_FIFO, SCHED_RR, etc.

param is a structure containing the priority

43 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Scheduling classes (3)

The priority can be set on a perthread basis when a thread is created :

Then the thread can be created using pthread_create(), passing the attr structure.

Several other attributes can be defined this way: stack size, etc.

struct sched_param parm; pthread_attr_t attr;

pthread_attr_init(&attr); pthread_attr_setinheritsched(&attr,

PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); parm.sched_priority = 42; pthread_attr_setschedparam(&attr, &parm);

44 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Memory locking

In order to solve the nondeterminism introduced by virtual memory, memory can be locked

Guarantee that the system will keep it allocated

Guarantee that the system has preloaded everything into memory

mlockall(MCL_CURRENT | MCL_FUTURE);

Locks all the memory of the current address space, for currently mapped pages and pages mapped in the future

Other, less useful parts of the API: munlockall, mock, munlock.

Watch out for noncurrently mapped pages

Stack pages

Dynamicallyallocated memory

45 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Mutexes

Allows mutual exclusion between two threads in the same address space

Initialization/destruction pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *mutexattr); pthread_mutex_destroy(pthread_mutex_t *mutex);

Lock/unlock pthread_mutex_lock(pthread_mutex_t *mutex); pthread_mutex_unlock(pthread_mutex_t *mutex);

Priority inheritance must explictly be activated pthread_mutexattr_t attr; pthread_mutexattr_init (&attr); pthread_mutexattr_getprotocol

(&attr, PTHREAD_PRIO_INHERIT);

46 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Timers timer_create(clockid_t clockid,

struct sigevent *evp, timer_t *timerid)

Create a timer. clockid is usually CLOCK_MONOTONIC. sigevent defines what happens upon timer expiration : send a signal or start a function in a new thread. timerid is the returned timer identifier.

timer_settime(timer_t timerid, int flags, struct itimerspec *newvalue, struct itimerspec *oldvalue)

Configures the timer for expiration at a given time.

timer_delete(timer_t timerid), delete a timer

clock_getres(), get the resolution of a clock

Other functions: timer_getoverrun(), timer_gettime()

47 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Signals

Signals are an asynchronous notification mechanism

Notification occurs either

By the call of a signal handler. Be careful with the limitations of signal handlers!

By being unblocked from the sigwait(), sigtimedwait() or sigwaitinfo() functions. Usually better.

Signal behaviour can be configured using sigaction()

Mask of blocked signals can be changed with pthread_sigmask()

Delivery of a signal using pthread_kill() or tgkill()

All signals between SIGRTMIN and SIGRTMAX, 32 signals under Linux.

48 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Interprocess communication

Semaphores

Usable between different processes using named semaphores

sem_open(), sem_close(), sem_unlink(), sem_init(), sem_destroy(), sem_wait(), sem_post(), etc.

Message queues

Allows processes to exchange data in the form of messages.

mq_open(), mq_close(), mq_unlink(), mq_send(), mq_receive(), etc.

Shared memory

Allows processes to communicate by sharing a segment of memory

shm_open(), ftruncate(), mmap(), munmap(), close(), shm_unlink()

49 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Debugging realtime latencies

50 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

ftrace Kernel function tracer

New infrastructure that can be used for debugging or analyzing latencies and performance issues in the kernel.

Developed by Steven Rostedt. Merged in 2.6.27. For earlier kernels, can be found from the rtpreempt patches.

Very well documented in Documentation/ftrace.txt

Negligible overhead when tracing is not enabled at runtime.

Can be used to trace any kernel function!

See our video of Steven's tutorial at OLS 2008: http://freeelectrons.com/community/videos/conferences/

http://free-electrons.com/community/videos/conferences/

51 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Using ftrace

Tracing information available through the debugfs virtual fs (CONFIG_DEBUG_FS in the Kernel Hacking section)

Mount this filesystem as follows: mount t debugfs nodev /debug

When tracing is enabled (see the next slides), tracing information is available in /debug/tracing.

Check available tracers in /debug/tracing/available_tracers

52 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Scheduling latency tracer CONFIG_SCHED_TRACER (Kernel Hacking section)

Maximum recorded time between waking up a top priority task and its scheduling on a CPU, expressed in µs.

Check that wakeup is listed in /debug/tracing/available_tracers

To select, reset and enable this tracer: echo wakeup > /debug/tracing/current_tracer echo 0 > /debug/tracing/tracing_max_latency echo 1 > /debug/tracing/tracing_enabled

Let your system run, in particular realtime tasks. Example: chrt f 5 sleep 1

Disable tracing: echo 0 > /debug/tracing/tracing_enabled

Read the maximum recorded latency and the corresponding trace: cat /debug/tracing/tracing_max_latency

53 Free Electrons. Kernel, drivers and embedded Linux development, consulting, training and support. http://freeelectrons.com

Useful reading

About realtime support in the standard Linux kernel

Internals of the RT Patch, Steven Rostedt, Red Hat, June 2007 http://www.kernel.org/doc/ols/2007/ols2007v2pages161172.pdf Definitely worth reading.

The RealTime Linux Wiki: http://rt.wiki.kernel.org “The Wiki Web for the CONFIG_PREEMPT_RT community, and realtime Linux in general.” Contains nice and useful documents!