D u k e S y s t e m s

CPS 110 210 310

Introduction to Operating SystemsFall 2013

Jeff ChaseDuke University

Resources to know about

• Course Web– http://www.cs.duke.edu/~chase/cps310

– Or other links through CS department

– All powerpoints, policies, reading, schedule, lab (project) instructions are posted there.

• Piazza– Announcements

– Chatter. “Maybe even anonymous posting.”

• Sakai– Minimal use, e.g., mainly for disseminating grades

Meetings

• “Lectures”:– MW 3:05 – 4:20

– ~25 lectures total

• Recitations– Fri 3:05, Bio Sci 111

– TA: Balakrishnan Chandrasekaran

• Two midterms– 10/4 (recit) and 11/6 (in class)

• “Light” final exam (1.5x)– 12/15 from 2-5 PM

“Bio Sci”

What is this course about?

• Programs

• Platforms

• Performance

• …

User Applications

Operating System

Substrate / Architecture

“The system is all the code your program uses that you didn’t have to write.”

http://steve-yegge.blogspot.com/2008/03/get-that-job-at-google.html

Steve Yegge

Course goals

• Learn to think about systems holistically.

• Sound mastery of structures and principles.

• Reinforce with practical, concrete examples.

• Minimize “unknown unknowns”.

2003

OS Platform: A Model

OS platform: same for all applications on a systemE,g,, classical OS kernel

Libraries/frameworks: packaged code used by multiple applications

Applications/services. May interact and serve one another.

OS platform mediates access to shared resources.

[RAD Lab]

“Software Architecture”

User Applications

Operating System(s)

Substrate / Architecture

Softwarearchitecture

Computerarchitecture

Comparative architecture: what worksReusable / recurring design patterns

• Used in OS• Supported by OS

Physics stops here.

Platform abstractions

• Platforms provide “building blocks”…

• …and APIs to use them to construct software.– Instantiate/create/allocate

– Manipulate/configure

– Attach/detach

– Combine in uniform ways

– Release/destroy

• Abstractions are layered.– What to expose? What to hide?

The choice of abstractions reflects a philosophy of how to build and organize software systems.

Managing Complexity

Environment

System

Component

Component

Component

Component

SystemSystems are built from components.

Operating systems define styles of software components and how they interact.

OS maps components onto the underlying machine.

…and makes it all work together.

Comparative software architecture

Large, long-lived software systems are like buildings.They are built by workers using standard design patterns.They depend on some underlying infrastructure.But they can evolve and are not limited by the laws of physics.

Watch it!

Computer systems is a liberal arts discipline.

Just because someone will pay you to do it doesn’t mean it’s not liberal arts.

Course prerequisites

• Basic data structures and programming– Lists, stacks, queues, graphs, DAGs, trees

– Abstract data types (ADTs), classes, objects

– Dynamic data structures

• Basic architecture– CPU context: registers

– Execution: runtime stack and frames

– Memory and L1/L2/L3 caches, DMA I/O

– Virtual addressing and memory layout

• Basic discrete math and probability

Dynamic data structures

A simple module

• A set of procedures/functions/methods.

• An interface (API) that defines a template for how to call/invoke the procedures.

• State (data) maintained and accessed by the procedures.

• A module may be a class that defines a template (type) for a data structure, which may have multiple instances (objects).

P1()

P2()

P3()

P4()

state

Abstract Data Type (ADT): the module’s state is manipulated only through its API (Application Programming Interface).

Code: instructions in memory

_p1: pushq %rbpmovq %rsp, %rbpmovl $1, %eaxmovq %rdi, -8(%rbp)popq %rbpret

load _x, R2 ; load global variable xadd R2, 1, R2 ; increment: x = x + 1store R2, _x ; store global variable x

A Peek Inside a Running Program

0

high

code library

your data

heap

registers

CPU core

R0

Rn

PC

“memory”

x

x

your program

common runtime

stack

address space(virtual or physical)

e.g., a virtual memory for a running program

(process)

SP

y

y

Data in memory64 bytes: 3 ways

p + 0x0

0x1f

0x0

0x1f

0x1f

0x0

char p[]char *p

int p[]int* p

p

char* p[]char** p

Pointers (addresses) are 8 bytes on a 64-bit machine.

Memory is “fungible”.

Heap: dynamic memory

Allocated heap blocks for structs or objects.

Align!

The “heap” is an ADT in a runtime library: the code to

maintain the heap is a heap manager.

It allocates a contiguous slab of virtual memory from

the OS kernel, then “carves it up” as needed.

It enables the programming language environment, to

store dynamic objects.

E.g., with Unix malloc and free library calls.

Free block

Read it on the course web

http://www.media-art-online.org/java/help/how-it-works.html

But Java programs are interpretedThey run on an “abstract machine” (e.g., JVM) implemented in software.

”bytecode”

Platforms are layered/nested

Grades: CPS 210 Fall 2012

4 A+

8 A

11 A-

13 B+

8 B7 B-

9 C* or lower

Cumulative Distribution FunctionCPS 2/310 Spring 2013

Total score X

Probabilitythat a random student’s score is X or below

A*

B*C*

10%quantile

90%quantile

median value

80% of the requests have response time R with x1 < R < x2.

x1 x2

“Tail” of 10% of requests with response time R > x2.

What’s the mean R?

Understand how the mean (average) can be misleading, e.g. if tail is heavy.

A few requests have very long response times.

50%(median)

Cumulative Distribution Function (CDF)

What is this course about?

• Programs

• Platforms

• Sharing

• Concurrency

• Storage

• Protection and trust

• Resource management

• Virtualization

• Scale and performance

• Abstractions

User Applications

Operating System

Substrate / Architecture

Reading• Course notes and slides

• External sources on every topic– OS in Three Easy Pieces

– A few academic papers and web readings

– Yes, even a “comic book”

We’ll look at these with varying levels of scrutiny.

http://csapp.cs.cmu.edua classic

Web/SaaS/cloudhttp://saasbook.info

Saltzer/KaashoekVery MITDo not buy kindle edition.

New!$10!

No text, but these may be useful.

New! $75!

Workload: projects

1. Dynamic heap memory (malloc/free)

2. Unix shell (“Devil Shell”)

3. Java concurrency: multi-threaded programming (“Elevator”)

4. Key/value store (“Devil Filer”)

5. Performance evaluation of storage server

Collaboration

• OK among groups:– General discussion of course concepts and

programming environment.

– “What does this part of the handout mean?”

• Not OK among groups– Design/writing of another’s program

– “How do I do this part of the handout?”

• Definitely not OK:– Using code from a previous semester.

• If in doubt, ask me.

Thoughts on cheating

Cheating is a form of laziness.

Cheating happens at those other schools.

Duke students work hard and don’t cut corners.

Your work is your own: if in doubt, ask.

Listen to what shame tells you.

Extra slides

The point of the remaining slides is:

• We take a broad view of “operating systems” encompassing a variety of application platforms.

• We start with Unix, a canonical/classical OS.

• Unix has continuing relevance: it continues to thrive deep inside the rich platforms we use today: knowing about the Unix kernel helps to understand how they work.

• Hardware and application demands change rapidly. Operating system kernels evolve slowly, but we often add more code around them to adapt to change.

• You’ll see these slides again.

What is this course about?

• “Greater Unix”– Classical OS abstractions and structure

– Systems programming with C and Unix

• Networked systems– Sockets and servers, smartphones to clouds

– Elementary cryptosystems

– Distributed systems topics

• Managing concurrency– Threads, multi-threaded Java

• Managing storage and data– Files, caches, disks, recovery, content delivery

Some lessons of history

• At the time it was created, Unix was the “simplest multi-user OS people could imagine.”– It’s in the name: Unix vs. Multics

• Simple abstractions can deliver a lot of power.– Many people have been inspired by the power of Unix.

• The community spent four decades making Unix complex again....but the essence is unchanged.

• Unix is a simple context to study core issues for classical OS design. “It’s in there.”

• Unix variants continue to be in wide use.

• They serve as a foundation for advances.

[http://www.android.com]

“Classical OS”

Reloaded.

Virtual Machine(JVM)

End-to-end application delivery

Cloud and Software-as-a-Service (SaaS)Rapid evolution, no user upgrade, no user data management.Agile/elastic deployment on virtual infrastructure.

Where is your application?Where is your data?Where is your OS?

SaaS platform elements

[wiki.eeng.dcu.ie]“Classical OS”

browsercontainer

OpenStack, the Cloud Operating SystemManagement Layer That Adds Automation & Control

[Anthony Young @ Rackspace]

EC2 The canonical public cloud

Virtual Appliance

Image

Canonical OS Example: “Classical OS”

• Unix/Linux, Windows, OS-X

• Research systems– Multics

– Mach

– Minix

– …

User Applications

Operating System

Substrate / Architecture

AggregationCompositionOrchestration

Exponential growthIncreasing diversity

Backward compatibility

Drivers of Change

Broad view: smartphones to servers, web, and cloud.

1

10

100

1000

10000

1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006

Pe

rfo

rma

nce

(vs

. V

AX

-11

/78

0)

25%/year

52%/year

??%/year

Moore’s Law and CPU Performance

From Hennessy and Patterson, Computer Architecture: A Quantitative Approach, 4th edition, 2006

Sea change in chip design: multiple “cores” or processors per chip

3X

Uniprocessor Performance (SPECint)