Hardware interrupts overview for Solaris X86

Hardware interrupts overview for Solaris X86

Welcome to OpenSolaris, and the wonders of Solaris 10.

This paper provides a brief introduction to hardware interrupts on x86 platforms. It is relevant for Intel and AMD based platforms. Interrupt handling is done via interrupt controller hardware in the system which are mostly sideband signals. Inband interrupts, for e.g. Message Signalled Interrupts (MSIs), introduced with PCI v2.2 specification onwards, will be discussed in another blog. MSIs are becoming mainstay with advent of new interrconnects like PCI-Express.

However, there are mainly two kinds of hardware interrupt controllers which are commonly used on x86 platforms:

1. 82c59(A) PIC(Programmable Interrupt Controller)

This is supported by the Solaris uppc(7d) module and its source is located at usr/src/uts/i86pc/io/psm/uppc.c.  Each PIC can handle 8 vectored priority interrupting sources and there are two PICs cascaded together to provide 16 interrupts on x86 systems.  However, one of the pin - IRQ2 of the 1st PIC is used to cascade the 2nd PIC and so there are only 15 interrupt sources.  This can not be used for multiprocessor (MP) systems without any major modifications.

2. APIC (Advanced Programmable Interrupt controller)

This is supported by the Solaris pcplusmp(7d) module and its source is located at usr/src/uts/i86pc/io/pcplusmp/apic.c.  It consists of two components - I/O APIC and Local APIC.  The Local APIC is embedded in the CPU while the I/O APIC is used for connecting the interrupting sources.  The Local APIC also has the capability to send interprocessor interrupt from one cpu to another and so APIC is widely used on all the x86 MP systems.  Each system can have multiple I/O APICs and each I/O APIC can have 4, 16, 20 or 24 interrupt pins.  Since the Local APIC is embedded in the CPU and the I/O APIC can handle more than 16 interrupt sources, even the single-CPU systems uses APIC as well instead of some other hardware.

There are many systems, which have I/O APICs with 4 inputs(this is typically done for PCI-X slotted systems, where each slot is given a dedicated I/O APIC, enabling INTA-INTD for each of the slots to have a dedicated input).

Solaris supports multiple I/O APICs.

Here is a diagram that shows the APIC on a two CPU system:-
 

             processor #1         processor #2
+-----------+ +-----------+
| | | |
| CPU | | CPU |
| | | |
+-----------+ +-----------+
| local APIC| | local APIC|
+-----------+ +-----------+
\^ \^
| |
| |
| |
v v Processor system bus
<----------------------------------------------------------------->
\^
|
|
|
+-----------------------------+
| | |
| v |
| +--------+ |
| | | |
| | Bridge | |
| | | |
| +--------+ |
| \^ |
| | |
| v PCI |
| <------------------> |
| \^ |
| | |
| v |
| +----------+ |
| | | |
| | I/O APIC |<-----|---External
| | |<-----|---Interrupts
| +----------+ |
| |
+-----------------------------+
System Chip set

Solaris x86 interrupt handling overview

When a device driver adding the interrupt through ddi_add_intr(9f), it eventually gets to uppc_addspl() in uppc(7d) for PIC or apic_addspl() in pcplusmp(7d) if using APIC.  The interrupt pin will be identified and then enabled.  It is quite simple for the uppc(7d) case, the interrupt pin to enable on the PICs is basically 1-1 mapped to the "IRQ#" or the "interrupt" property of the device on Solaris.  But for APIC (pcplusmp(7d)), it is a lot more complicated as internally it either uses the MP Spec. 1.4[2] or ACPI specification[3]  to locate the right interrupt pin of the right I/O APIC for the device. The system BIOS sets up how the interrupts are routed and saves that information in either the MP Specification table or somewhere that ACPI can easily access. pcplusmp(7d) then access that information to initialize and add the interrupts.

During the ddi_add_intr(9f) call, the device interrupt handler's entry point is stored in the autovect[] and the interrupt pin will be enabled through uppc_addspl (for PCI) or apic_addspl(for APIC).  Also, before the interrupt pin is enabled, an interrupt "vector" (refered to as "vector" from now on) will have to be selected for the CPU to trigger when that particular interrupt comes in.  For Intel CPUs, there are total 256 vectors and the first 32 vectors are reserved for special functions and so the first available vector for devices is 32 (or hex 0x20). 

For uppc(7d), vectors are set up such that the 1st pin or IRQ#0 is mapped to 32, IRQ1 to 33 and so on.  As for pcplusmp(7d), it is not as simple.  Solaris handles interrupts based on interrupt priority and each device is assigned a unique priority (can be modified by the device driver).  Say, if a device "abc" is assigned  priority 5, then all other interrupts at 5 or lower can NOT be triggered when the interrupt handler of "abc" is executing.  However, an interrupt of priority 6 or higher is allowed to trigger.  Since APIC has mechanism to prioritize interrupts, pcplusmp(7d) needs to select the vectors accordingly.

In order to handle the interrupt priority properly, there are few internal interface calls provided by uppc(7d) and pcplusmp(7d). They are the uppc_intr_enter()/uppc_intr_exit() for the uppc(7d); and apic_intr_enter()/apic_intr_exit() for pcplusmp(7d).  After the interrupt is triggered but before the interrupt handler is called, uppc_intr_enter() or apic_intr_enter() will be called to setup the interrupt priority accordingly to block all other interrupts with the same or lower priority.  After the interrupt handler is completed, then uppc_intr_exit() or apic_intr_exit() is called to restore the interrupt priority.

On the x86 platform, all the local variables of the interrupt handler are on stack. Also, if the interrupt handler needs to call another function, the parameters that are passed to the function are on stack too. i.e. all the interrupt handlers should use the stack one way or the other.

Solaris code that handles interrupts

Below are few code snippets that deal with interrupts.

To begin with the 256 vector entries are defined in the autovect[] table shown below:
usr/src/uts/common/io/avintr.c

#define MAX_VECT 256
struct av_head autovect[MAX_VECT];

usr/src/uts/common/sys/avintr.h

struct autovec {

/\*
\* Interrupt handler and argument to pass to it.
\*/
struct autovec \*av_link; /\* pointer to next on in chain \*/
uint_t (\*av_vector)();
caddr_t av_intarg;
uint_t av_prilevel; /\* priority level \*/

/\*
\* Interrupt handle/id (like intrspec structure pointer) used to
\* identify a specific instance of interrupt handler in case we
\* have to remove the interrupt handler later.
\*
\*/
void \*av_intr_id;
dev_info_t \*av_dip;
};

av_vector is the device interrupt handler.

struct av_head {
struct autovec \*avh_link;
ushort_t avh_hi_pri;
ushort_t avh_lo_pri;
};

- All interrupts run at some priority which has a ceiling of LOCK_LEVEL.
Interrupts below LOCK_LEVEL run as threads.

usr/src/uts/intel/sys/machlock.h

#define CLOCK_LEVEL 10
#define LOCK_LEVEL 10

- The following sequence shows what is done for each CPU to allocate
 enough interrupt threads to handle the interrupts. Since interrupts
are prioritized, one interrupt thread per priority should be sufficient.

usr/src/uts/i86pc/os/mp_startup.c

void
start_other_cpus(int cprboot)
{
for (who = 0; who < NCPU; who++) {
mp_startup_init(who);
...
}

void
mp_startup_init(void)
{
init_intr_threads(cp);
...
}

usr/src/uts/i86pc/os/intr.c

/\*
\* Allocate threads and stacks for interrupt handling.
\*/
#define NINTR_THREADS (LOCK_LEVEL-1) /\* number of interrupt threads \*/

void
init_intr_threads(struct cpu \*cp)
{
int i;

for (i = 0; i < NINTR_THREADS; i++)
(void) thread_create_intr(cp);
...
}

usr/src/uts/common/disp/thread.c

void
thread_create_intr(struct cpu \*cp)
{
- Here is the actual code handling the interrupts on x86.
\*setlvl is the wrapper for uppc_intr_enter() or apic_intr_enter().
}



NOTE: Calling the interrupt handler is done in low level assembly code
which is not discussed here.

Reference:

1.See Chapters 5 and 7 of the Intel Architecture Software Developer's Manual Volume 3: System Programmer Guide for details on how interrupts work on the x86 platform.
2. Intel Multi-Processor Specification v1.4
3. Advanced Configuration & Power Interface (ACPI) specification home


PS: Lots of thanks to Johnny Cheung, also in Solaris I/O, for originally contributing to this material.

Technorati Tag: Technorati Tag:
Comments:

Thanks for posting this article.

I have been learning about the interrupt handling mechanism for my course and this has given me a further area of research on the subject.

I didn't know there were two different types of of hardware interrupt controllers, as I figured all were produced via a single piece of hardware.

Posted by Binary Soldier on July 31, 2009 at 11:41 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

anish

Search

Top Tags
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today