aas white paper

8/3/2019 Aas White Paper

http://slidepdf.com/reader/full/aas-white-paper 1/14

hp -ux / v irtual

memory

May 2003

technical

white pap er

Adap tive Add ress Spac e

tab le of contents introduction 2exec utive summa ry 2background 3limita tions o f trad itiona l HP-UX ad dress spa ce mo dels 6wha t do es MPAS p rovide? 7

larger, mo re homo ge neous ad d ress spa c e 7mma p independe nce 9multiple mm ap ()s of the sam e da ta 9preffered ad dress for shma t() 9

ho w to get MPAS? 9usage deta ils 9a lternat ives to MPAS 12performance 12c om pa tibility/ interop erab ility 12configuration 12glossary (simp lified ) 13for mo re informa tion 13



Adaptive Ad dress Spa ce

introduction

executive

summary

The Ada p tive Add ress Spac e (AAS) fea ture is introd uc ed in HP-UX 11i V2. AAS

prov ides a new a ddress sp ac e layo ut, MPAS (Mo stly Priva te Add ress Spa ce),

which helps applications targeted for HP-UX running on Intel® Itanium ®

Proc essor Family (IPF) ma chines. MPAS c a n a lso b e used to a id a pp lica tion

p orta b ility to HP-UX.

This pap er de sc ribe s the fea tures AAS p rovides, how a pp lic a tions c an use the sefeatures and impa c ts thereof.

The me mo ry mo del trad itiona lly used by HP-UX trade s off a n a pp lica tion’s

flexib ility in using its address spac e fo r pe rforma nc e. This mea ns tha t 32-b it

a pp lica tions cod ed for HP-UX tha t share a lot of da ta amo ng p rocesses enjoy

c ertain performa nc e ad van tag es that they would not on other op era ting

systems. Howe ver, the se a pplica tions a re restric ted by c erta in limita tions o f HP-

UX’s me mo ry mo d el.

AAS removes a ll suc h limita tions from HP-UX for app lic a tions tha t c hoose to use

the fe a ture. The remo val of the se limita tions mea ns tha t ap plic a tions have

greate r control of their own a dd ress spa c e and have me mo ry-manag eme nt

fea tures tha t d id not exist on HP-UX but were p rovide d by several o ther

operating systems (e.g. Solaris® and Linux®). This aid s po rtab ility o f ap plica tionsto HP-UX from these o ther op erating systems. Howe ver, this ma y c ause the

a pp lication to loose som e of the p erformanc e ad vanta ge s typica lly enjoyed

b y HP-UX ap plica tions.

App lica tion vend ors cod ing for HP-UX ca n a lso use the flexibility a nd features

offe red b y AAS to simp lify the ir d esign.

This app lies only to HP-UX a pp lica tions comp iled and linked for IPF p latfo rms.

2



Adaptive Ad dress Spa ce

background HP-UX prov ides va rious a dd ress spa ce layo uts for ap p lica tions to c hoose from.

An a d dress sp ac e layo ut rep resents a p roc ess’s a cc essible virtua l ad dress

spa ce a nd how the o perating system d ivides it into different reg ions into which

different types of mem ory objects c an b e a ttac hed.

The default address spac e layout fo r 32-b it proc esses is known as SHARE-

MAGIC. In this layout, the first 1GB o f the proc ess’ s ad dress spac e is reservedexc lusively for text. This 1GB is shared by a ll proc esses exec uting the same text

(henc e the na me SHARE-MAGIC). The me mo ry layou t fo r a 32-bit SHARE-

MAG IC p roc ess looks like:

(The dia g ram show s a SHARE-MAGIC p roc ess on IPF. Note , the fig ure is not d rawn to

scale. The m aximum sta ck sizes are de c ided b y system tuna bles and p rocess resourcelimits at the t ime of p roce ss sta rt-up) .

Notice tha t the add ress sp ac e o f the p roc ess is sp lit into 4 qua drants,

numbered 0-3. Ea ch quad rant is 1GB in size. Quadrant 0 is reserved for text,

qua drant 1 for proc ess priva te da ta and qua d rants 2 and 3 for data that ca n

be shared between processes.

The 2GB of sha red d a ta spa c e b etwe en v irtual addresses 0x8000 0000 and

3



0xFFFF FFFF (q ua drants 2 and 3) is consum ed b y most 32-bit p roc esses in the

system. For example, if p roc esses A and B want to share 1MB of d a ta , they w ill

use this sp ac e to share it (the kernel will p ick a v irtua l ad dress in this rang e). This

1MB d a ta ma y not b e visible to p roc ess C, but the d a ta will consume 1MB of

virtual ad dress sp ac e tha t proc ess C c ould hav e used . Proc esses A and B will

a c c ess the d a ta using the same virtual a dd ress – henc e they will read / write

the same da ta . This is how HP-UX a llow s multiple p roc esses to sha re d a ta – by

giving them the same virtual address (i.e. aliasing of different virtual addresses

to the sam e physical pa ge is not need ed ).

Also no te tha t this layout reserves 1GB for te xt. This 1GB may not be used by the

proc ess for any other purpo se. All proc esses using the sa me b inary with the

default layout will share the text using the same virtual address.

This me tho d o f sha ring c osts virtual ad d ress spa c e, bu t is very effic ient b ec ause

of the lac k of aliasing. The a dvanta ge s of no t ha ving a liasing a re:

1. Fewe r faults a re need ed b y the proc esses to ac c ess the data .

2. The sha ring p roc esses incur fewe r TLB misses when ac cessing the d ata .

3. Less sp ac e is need ed by the kerne l for its da ta structures.

Som e a pp lica tions ma y no t like this pa rticula r distribution o f the qua drants. For

instanc e, an a pp lic a tion may ne ed more tha n 1GB of p roc ess private ad dress

spa c e (for examp le, b ec ause it has a large hea p). Or, it ma y wa nt mo re than

2GB of shared ad dress sp ac e. And so on.

HP-UX provid es other address spa c e layouts. The 32-b it layo uts of HP-UX a re

summ arized in the ta b le below:

ta ble 1a . 32-bit Ad d ress Spac e layo uts.

Text

Process

private

address

space

Shared

address

space

Limitations on

usage

Defa ult (Sha re-

Magic)Quad rant 0 Quad rant 1

Quadrants

2,3None.

Exec-Mag ic Qua dran ts 0,1Quadrants

2,3

None.

Shmem -Magic Qua drant 0Quadrants

1,2,3None.

Q3 Private

Share-MagicQua drant 0 Qua drants 1,2 Qua drant 3

PA-RISC

p la tforms only

Q4 Private

Share-MagicQuad rant 0

Quadrants

1,2,3None.

PA-RISC

p la tforms only

Q3 Private

Exec-MagicQua dran ts 0,1,2 Qua drant 3

PA-RISC

p la tforms only

Q4 Private

Exec-MagicQuadrants 0,1,2,3 None.

PA-RISC

p la tforms only

MPAS Qua drants 0,1,2,3Availab le o nly

with AAS

64-b it pro c esses a re trea ted d ifferen tly on IPF and PA-RISC. On IPF, the a d dress

spa ce for a 64-bit proc ess is divided into 8 oc tan ts. The layo ut fo r the d efault 64-

4



b it proc ess is shown b elow . This typ e o f la yout is ca lled MGAS (for Mo stly Glob a l

Add ress Sp ac e). The layout is:

(The diag ram show s a 64-bit MGAS proc ess on IPF. Note, the figure is not draw n to sc ale.The m aximum sta ck sizes are d ec ided by system tuna bles and p rocess resource limits atthe time of process start-up).

For a 64-bit proc ess, text can oc cupy up to 1 oc tant . This entire o c tant is not

ava ilab le to the a pp lic ation for any other use.

64-bit MGAS app lica tions share da ta a mong themselves by using the sa me

me c han ism a s 32-bit app lica tions (i.e. the d a ta is a tta c hed to the shared virtua l

a ddress spac e, and a ll sharing p roc esses ac c ess it using the sa me v irtua l

a ddress. This eliminates the need for a liasing of virtual a ddresses. But shared

virtual a dd ress consumed by one p roc ess will imp ac t o ther proc esses). Shared5



lim ita tions of

traditional HP

UX address

spa c e mod els

virtua l ad dress sp ac e fo r 64-bit MG AS p roc esses lies in oc ta nts 0,1 and 6.

The layou t fo r a 64-bit proc ess on PA-RISC is significa ntly d ifferen t and is no t

shown here.

The trad itiona l ad dress spa c e layouts o f HP-UX suffered from the fo llowing

limitations:

1. On HP-UX, proc esses have a fixed d istribu tion o f the p roc esses’ virtua l

ad dress spa c e into ‘ shared’ ad dress spa c e a nd ‘ private ’ ad dress spa c e

(i.e. virtua l ad dress spa ce in which proc esses c an share d a ta a mong

them selves VS virtual a ddress spa c e in w hich proc esses ca n a tta ch

da ta tha t is meant for the use b y the a tta ching proc ess a lone). This

d istribution o f virtua l ad dress spac e is fixed a t c omp ile / link time. This

mea ns that a proce ss c annot d ynam ica lly dec ide how much da ta in

me mo ry it wants p rivate and how muc h it wants to share. This restriction

ma inly impa c ts 32-bit processes.

2. A 32-bit proc ess ca nnot g et a single d ata ob ject greate r than 1GB in

size if it wa nts to sha re it with o ther proc esses. This restric tion a pplies

eve n if the p roc ess has mo re than 1GB of shared a d dress spac eavailable.

3. The shared a d dress spac e a va ilable to 32-bit proc esses is consumed by

a ll p roc esses tha t read / write shared d ata . This means tha t a proc ess’ s

po ol of shared da ta spa c e availab le c an b e c onsumed by other

p roc esses – even by other proc esses tha t are no t sharing da ta with this

process.

4. The m ma p (2) system ca ll canno t b e used to m ap a po rtion o f a file

multiple times. A numb er of app lica tions and lib ra ries use the mm ap ()

system c all to read / write d a ta from files. The inab ility to ma p p ieces of

the file multiple time s c omplica tes ap p lica tion log ic. This ap plies to b oth

32-b it and 64-bit app lica tions. (This app lies only if the MAP_SHARED flag

is sp ec ified in the mma p() ca ll).

5. Some comp lex seq uenc es of mma p (2) ca n fail. For instanc e, if proc ess

A ma ps pa ge 1 of a file using the MAP_SHARED flag o f mmap (2), and

proc ess B tries to ma p p age s 1 and 2, it c ould fail. [The w orkaround in

this ca se is to hav e p rocess B ma p first, or have p roc ess A map bo th

page s 1 and 2 eve n thoug h it needs only 1 pa ge – proc ess A do es not

loose a ny virtua l ad dress spa ce in d oing this]. This app lies to both 32-bit

and 64-bit applications.

The Ad ap tive Add ress Spa ce p rojec t provides the user with a new type of

address spa c e layo ut, c alled “ Mostly Priva te Ad dress Spac e” (or MPAS forshort), tha t c an overcom e these limita tions. This makes it ea sier to port

a pp lica tions to HP-UX from other op erating system s tha t p rovide these fea tures.

It a lso simp lifies the d esign o f a pplica tions written for HP-UX.

6



what does The MPAS layo ut p rov ides the following fea tures:

MPAS

provide?

large r, more The address spa c e layout fo r 32-bit MPAS p roc esses loo ks like:homogeneous address space

(Note tha t the figure is not d raw n to scale. The m aximum sta c k sizes a re d ec ided ba sedon system tuna bles and p rocess resource limits at the t ime o f p rocess sta rt up ).

In this layo ut the entire 4GB, i.e. all 4 qua d ran ts, are a va ilab le for the p rocess to

c onsume in any ma nner it choo ses. No o ther process c onsumes any part of this

p roc ess’s a dd ress spa c e. Private and shared d ata c an b oth b e a ttac hed a t

a ny loc a tion. A proc ess is not d isa llow ed to a tta c h ob jec ts grea ter than 1GB insize. This gives the p roc ess mo re flexibility in how it c onsumes its ad dress spac e.

However this sc heme implies that d ata ca n b e shared b etwe en two p roc esses

only b y a liasing their virtua l ad dresses to the sa me p hysica l pa ge . This lead s to

some performance inefficiencies.

7



The layou t fo r 64-bit MPAS proc esses is shown below:

(Note t hat the figure is not d rawn to scale. The m aximum sta c k sizes are d ec ided ba sed

on system tuna bles and p rocess resource limits at the t ime o f p rocess sta rt up).For 64-bit MPAS proc esses, text ca n oc c upy o ne en tire o c tan t of a dd ress

spa ce. This oc tant is no t ava ilab le to the p roc ess for any o ther use. All

p rocesses running the sa me b inary use sha re the te xt using the same virtual

a ddress – without need for a liasing v irtua l ad dresses.

A 64-bit MPAS proc ess can c onsume a d dress spac e from o c tant 6 only by

prov iding spec ial instructions to the o perating system – this a dd ress spa ce is not

c onsumed by MPAS a pp lica tions by de fault.

8



mmap For MPAS processes, mmap() ca lls ma d e b y this p roc ess are indep end ent o f

independence ca lls ma de b y a ny othe r p roc ess. In the ca se c ited e arlier in the sec tion

“ limita tions o f traditiona l HP-UX address spa c e mod els” , item numb er 5, p roc ess

B c ould not mma p() p ag es 1 and 2 of a file bec ause a nother proc ess A had

mm ap()ed pa ge 1. This, and all suc h inter-p roc ess limitations a re remove d for

MPAS p roc esses.

multiple mm ap ()s As described in limitation number 4 in the section “limitations of traditional HP-

of the sam e data UX ad dress spac e models” , trad itiona l HP-UX proc esses (those no t using MPAS)

ca nnot mmap the sa me p ortion of a file more than onc e using the

MAP_SHARED flag . For instanc e, if a p roc ess has ma pped page 1 of a file, then

it ca nnot ma p p ag e 1 a ga in (or ma p p a ges 1 and 2) without first unmap ping

the p revious ma pp ing.

For an MPAS proc ess, eac h ca ll to m ma p is indep end ent of o ther mapp ings

ma de by the sa me p roc ess. Apa rt from the a bo ve me ntioned a bility to ma p

the same p ortion o f the file multip le time s, this a lso mea ns tha t ea ch ma pp ing

can be mprotect(2)’ed individually.

preffered add ress Trad itiona lly, HP-UX ap plica tions ha ve a ve ry limited ab ility to spec ify a

for shmat() preferred a ddress during the c all to shma t(2) to a ttac h a system V shared

memory segment. A process either specified a NULL address, in which case the

kernel wa s free to c hoo se a ny address. Or, the p roc ess had to sp ec ify the verysam e a dd ress that o ther proc esses had atta c hed the segm ent a t. Any other

a ddress would fa il. This restrict ion is lifte d for MPAS proc esses, whic h can spec ify

a ny p referred a dd ress with shma t().

how to get To use the fea tures provided by the AAS, the b inary has to b e c onve rted to use

MPAS?the MPAS layo ut. This c an be done under the fo llow ing rules:

• Binary ha s to be linked with a linker p rovide d with HP-UX 11i V2 or la ter.

• Provide ld(1) / cha tr(1) option to c hang e a n executa ble to MPASmodel: +as mpas

Other +as options:

• share_magic

• exec_magic

• shmem_magic

• mgas (sam e a sshare_magic for 32-bit)

(This sec tion mentions technica l deta ils that ca n b e skipp ed by a read er interested only in anusage d eta ilsove rview of AA S).

While d esigning / running a pplic a tions tha t use MPAS proc esses, the fo llowing

points will nee d to b e c onsidered :

1. An MPAS proc ess can g et up to 4GB of a d dress spa c e. To a c tually

succ eed in large a lloc ation a ttemp ts, the user would ne ed to set

tuna bles, proc ess resourc e limits, etc . to g et a ll the m em ory desired . E.g.

to get a 4GB hea p for a 32-b it MPAS proc ess, set the RLIMIT_AS limit a nd

9



the maxdsiz tunable appropriately, and ensure that enough swap is

available.

Similar c onstraints app ly if othe r typ es of la rge ob jec ts a re de sired – e.g.

if a la rge sta c k was nee ded , then the maxssiz tunab le and the

RLIMIT_STACK limit would hav e to b e a d justed instead of the ma xdsiz

tuna ble a nd RLIMIT_AS limit respec tively.

The ma n p age for setrlimit(2) is a go od sta rting p oint for informa tion onresource limits. The ma n p ages for the tuna bles ma xssiz(5) and

ma xdsiz(5) are go od starting points for information on som e o f the

address spa c e related tunables.

2. HP-UX on PA-RISC provides two types of layouts for binaries: q3private

and q4priva te . HP-UX PA-RISC b inaries using these layouts were not

sup ported on IPF machines running HP-UX. On HP-UX 11i V2 onwards,

they will be sup po rted , but w ill be t reated a s MPAS bina ries on IPF

platforms.

3. HP-UX prov ides the “ me mo ry window s” func tionality to 32-bit

app lica tions tha t need m ore co ntrol over the usage o f their shared

address spa c e. Since 32-bit MPAS proc esses ha ve the maximum

possible limit o f 4GB of a ddress spac e tha t c an b e used for shared

ob jec ts, MPAS processes should no t ha ve n eed to use the mem ory

windo ws func tionality. In fa c t, co mb ining MPAS p roc esses and me mory

windows is fraught with co mp lica tions and is not rec om mend ed .

If such a combina tion is unav oida b le, then the fo llow ing points should

be bo rne in mind:

o MPAS p roc esses can ma p ob jec ts not p resent in their memory

windows

o Howev er, MPAS p roc esses c reate ob jects in the ir own me mo rywindows. I.e. if Proc ess A (MPAS) a nd proc ess B (non-MPAS, say,

sha re mag ic) wa nt to share an ob jec t. Then either:

i. Process B should c reate the ob jec t (e .g. shmg et() w ith

IPC_CREAT)

ii. Or, Proc esses A and B shou ld b e in the same memory

window

iii. Or, Process A sho uld d o the shmget() w ith the

IPC_GLOBAL fla g

More informa tion on m emo ry windows c an b e ob tained from the link

provide d in the sec tion “ for more information”.

10



4. The MAP_GLOBAL and MAP_ADDR32 fla gs of m ma p (2) b ehave differently

fo r MPAS and non-MPAS p roc esses.

MAP_GLOBAL MAP_ADDR32 MAP_ADDR32|MAP_GLOBAL

32-b it

MPAS

No effect No effect No effect

32-bit,non-

MPAS

Go es to 4thquadrant

No effec t Sa me a sMAP_GLOBAL

64-b it

MPAS

Go es to 6th

Octant

No effect Sa me a sMAP_GLOBAL

64-bit,

non-

MPAS

Go es to 6th

Octant

Go es to

virtual

address <4GB

Go es to v irtua l ad dress

b etwe en 3GB and 4GB

5. MPAS p rocesses a tta ch shared library text in p riva te a ddress spac e.

Howe ver, the text is still sha red w ith other proc esses. Henc e, b reakpo ints

c anno t b e p lac ed in them . To p lac e b reakpoints in shared library text,use chatr +dbg enable.

1



alternatives to Rea sons for not using the MPAS layo ut c ould include:

MPAS 1. App lic a tion is ta rgeted for the PA-RISC a rchitec ture.

2. App lic a tion is ta rgeted for a version of HP-UX p reced ing HP-UX 11i V2.

3. App lic ation vendo r wa nts to retain pe rforma nce a dva ntag es provide d

to non-MPAS app lica tions.

App lica tions that c anno t use MPAS c a n ge t som e o f the a dva nta ges that

MPAS layouts p rovide by choo sing one of several ad dress spa ce layo uts

p rov ided by HP-UX as desc rib ed ea rlier.

Limita tion numb er 3 in the sec tion “ limitat ions o f trad itiona l HP-UX address

spa ce mod els” ca n b e a dd ressed b y using the memo ry-windows functionality.

performance Using MPAS layo uts ma y incur co sts in terms of performa nc e. The p erformance

cost c om es ma inly from using a liases. This performa nc e c ost b ec om es a fac tor

if the ap plica tion ha s a large a mount o f da ta tha t is shared b etween

p roc esses.

On the o ther hand , 32-bit proc esses using MPAS layo uts ga in the adva ntage of

a flexib le virtua l ad dress spac e. This flexibility translate s to a performa nc e

a dva ntag e to processes that can use larger add ress spa ce to do more wo rk,

faster. For exam ple, the Java Virtual Mac hine is very sensitive to the a mo unt o f

heap spa ce it has. Since MPAS a llow s proc esses to ha ve up to 4GB of a d dress

spa c e for the heap , this c an translate to a p erformanc e ad vanta ge.

The p erformanc e difference thus seen will va ry from a pp lica tion to ap plica tion.

The following fac tors can a id in dec iding w hether or not to use the fea ture. In

terms of p erforma nc e, MPAS should b e a go od alternat ive fo r:

o proc esses who se p erforma nce is sensitive to the a mo unt of p rivate da ta

spa c e availab le.

o proc esses that do a lot o f mp rotec t(2) op erations on shared d a ta

MPAS could lowe r p erforma nc e for app lica tions tha t:

o sha re a lot o f da ta b etwe en p roc esses

compatibility/ AAS provides b inary and API c ompa tibility for ap plica tions tha t do not use the

feature.interoperability

configuration An a pplica tion tha t wa nts to use MPAS layou ts need s to link with a spec ia l flag.

Deta ils a re ment ioned in the sec tion “ho w to ge t MPAS?”

12



glossary

(simplified)

for more

information

AAS: Ada p tive Ad d ress Spac e. The fea ture being d iscussed in this p ap er.

Aliasing: In this pap er, alia sing refers to the cond ition whe n two or mo re unique

virtua l ad dresses translate to the same physica l ad dress. All aliased virtua l

a dd resses c an b e used to a c cess the sa me da ta.

BSS: Bloc k Sta rted by Symb ol. The sec tion o f a p rogram’ s da ta which is used to

store g loba l da ta tha t is no t initialized explic itly by the p rog ramm er. (Imp licitly

initialized to 0).

MGAS: Mostly Glob al Ad dress Spac e. This is the defa ult address spac e layout o n

HP-UX.

MPAS: Mo stly Private Address Spac e. This is the new typ e o f a dd ress spac e

layout tha t is introd uc ed by the AAS p rojec t.

RISC: Red uc ed Instruc tion Set Comp uter. A com puter architec ture that red uc es

chip complexity by using simpler instructions.

RSE: Reg iste r Sta ck Engine. Trad itiona l proc essor arc hitec tures require spilling

and filling o f registers during func tion c a ll / return. On newe r, RISC a rchitec tures,

a register sta ck engine avoids this via c om piler co ntrolled rena ming of g eneral

reg isters. For de ta ils, refe r to the IA-64 Architec ture Softwa re Develop er’s

Manual.

TLB: Translat ion Look-a side Buffer. A sma ll table in the p roc essor’ s Memory

Mana ge me nt Unit tha t con tains translations from virtual a dd ress to p hysica l

a ddresses.

For mo re informat ion on memory-window s, go to

http://docs.hp.com/hpux/onlinedocs/os/memwn1_4.pdf

For mo re informat ion on the linker and other developer tools, go to

http :// doc s.hp.co m/ hpux/d ev/ ind ex.html# Develop er%20Too ls%20and%20Librari

es

For more information o n the IPF architec ture, see the Intel® IA-64 ArchitectureSoftware Developer's Manual.

For more information on the following, see the relevant HP-UX manual pages:

• mmap(2)

• mprotect(2)

• shmat(2)

• shmget(2)

• ld(1)

• cc(1)

• chatr(1)

13



• setrlimit(2)

• maxdsiz(5)

• maxssiz(5)

Intel® Itanium ® Proc essor Fa mily is a trad emark of Inte l Corpo ration in the US a nd o ther co untriesand is used und er lice nse.

Sola ris® is a registered trad em ark o f Sun Mic rosystems, Inc .

Linux® is a registered trad em ark of Linus Torva lds.

HP-UX and PA-RISC a re registered trad emarks of Hew lett-Packa rd c omp any.

The information in this do cu ment is subjec t to c hang e without notice .

© Cop yright Hewlett-Pac kard Co mpa ny 2003

05/2003

Publica tion numb er 1.0

14

aas white paper

Documents