Exploring the Dynamics of Self-Organising Systems withStochastic π-Calculus: Detecting Abnormal Behaviour in MAS

Luca Gardelli∗, Mirko Viroli, Andrea OmiciniDEIS, Alma Mater Studiorum—Universita di Bologna

via Venezia 52, 47023 Cesena, Italy{luca.gardelli, mirko.viroli, andrea.omicini}@unibo.it

Abstract

The intrinsic complexity of self-organisingMASs (multi-agent systems) makes it diffi-cult to predict global system evolutions atearly stages of the design process. Simulat-ing high-level models to analyse propertiesof a MAS design can anticipate detection ofincorrect / wrong design choices, and allowtuning of system parameters.In this paper, we take abnormal-behaviourdetection as a case study, and devise anartifact-based MAS architecture inspired byprinciples of the human immune systems.We use stochastic π-calculus to specifyand run quantitative large-scale simulations,which allow us to verify the basic applicabil-ity of our IDS (intrusion detection system)and possibly obtain a preliminary set of itsmain working parameters.

1 Introduction

The trend in today information systems engineering istoward an increasing degree of complexity and open-ness, leading to rapidly-changing requirements andhighly-dynamic environments. Furthermore, the in-creasing costs of systems engineering [Horn, 2001] callfor new engineering methodologies and tools. In thatsense social and natural sciences are recognised as richsources of inspiration: for instance, the AutonomicComputing initiative tries to face complexity by ap-plying self-regulating mechanisms typical of biologicalprocesses [Kephart and Chess, 2003].

Self-organisation is a promising theoretical frame-work to reduce complexity of systems engineering. Asystem is said to be self-organising if it is able to re-organise itself upon environmental changes, by localinteraction of its parts without any explicit pressurefrom the outside [Heylighen, 2003]. A system builtaccording to this principle is usually able to performcomplex tasks even though its components are far sim-pler when compared to a monolithic solution.

∗This work is partly supported by the European ScienceFoundation (ESF) Minema Scientific Programme, GrantNo. 805.

In this paper we elaborate along the line developedin [Gardelli et al., 2005b; 2005a] by exploring method-ological aspects of the engineering of self-organisingMASs. Because of the complexity inherent in suchsystems, as well as the difficulties in predicting theirbehaviour and properties, we find it crucial to exploitformal tools for simulating systems dynamics at theearly stages of design, in order to (i) nurture evolvingideas and design choices, and (ii) to provide betterfoundation for the feasibility of the system. In the de-ployment stage, the models devised can then providesuitable parameters for (iii) tuning the system.

Among the various formal models to specify quan-titative aspects of MASs we promote the use of thestochastic π-calculus process algebra [Milner et al.,1992; Priami, 1995]. Process algebras have alreadybeen successfully applied to several domains [Baeten,1990]: now their value is even more recognised infields like security [Abadi and Gordon, 1997], perfor-mance analysis [Gilmore and Hillston, 1994], socialinsects [Tofts, 1992], molecular biology [Regev et al.,2001], but still almost unexplored in the context ofself-organising and autonomic systems engineering.

In this paper we show that the π-calculus processalgebra can be fruitfully applied in particular to theengineering of multi-agent systems (MASs): in partic-ular we will show how to track the dynamics of globalsystem properties via stochastic simulations, and de-termining suitable system parameters.

In this paper, we apply these ideas to the studyof an intrusion detection (ID) infrastructure for openMASs: the focus is on detecting anomalies in agentsbehaviour drawing inspiration from principles of thehuman immune system [Forrest et al., 1997]. The in-frastructure we devise is based on the TuCSoN tech-nology [Omicini and Zambonelli, 1999]1. This allowsus to structure a MAS not only in terms of agents, butalso with tuple centres [Omicini and Denti, 2001] ascoordination artifacts [Omicini et al., 2004] and agentcoordination contexts (ACC) [Omicini, 2002]. Coor-dination artifacts are used to model resources in theenvironment on which agents act upon. ACCs spec-ify and enact the access policies which each agent issubject to, and can be used to both (i) reify relevantinformation about the agent/artifacts interaction, and

1http://tucson.sourceforge.net

(ii) to deny malicious agents access to the resourcesavailable in the environment.

To evaluate the impact of different design choicesand parameters of the ID infrastructure—such as in-spection/detection rates, number of inspectors, andthe like—we simulate the system behaviour in dif-ferent scenarios using SPiM specifications. SPiM—Stochastic Pi Machine—is a simulator that is able toexecute systems specifications written in stochastic π-calculus and track the quantitative system dynamics[Phillips, 2005].

The rest of the article is structured as follows. InSection 2 we briefly highlight the subject of ID andrelated mechanisms in the human immune system. InSection 3 we describe our reference architecture for aMAS based on TuCSoN, and show how to develop ananomaly detection application based on artifacts. Sec-tion 4 motivates the use of π-calculus and its stochas-tic extension. Then Section 5 discusses simulationsrelated to our case study, along with some meaning-ful experimental results. Finally, Section 6 concludeswith final remarks, and with some hints on the maindirections for our future research.

2 Intrusion Detection and the HumanImmune System

There are several mechanisms used to protect infor-mation systems, but usually only the basic ones areimplemented: (i) authentication—identity is provedby the knowledge of a secret (e.g. password) or a phys-ical unique property (e.g. fingerprint, retina, voice);(ii) authorisation—user actions on the system are con-strained by its role and the policy linked to that role.

However, application flaws typically cause thesemethods alone not to be sufficient [Forrest et al.,1997]. Furthermore, authorisation policies cannot ac-count for all possible sequences of actions, and a spe-cific sequence might exhibit unexpected side-effects: itis in general too expensive and impractical—or evenunfeasible—to intercept all emergent harmful paths atdesign-time.

Hence, automated tools are essential to enable run-time detection of malicious behaviours: in this direc-tion, many efforts have already been spent in devel-oping IDSs. An IDS tries to detect intruders and mis-use of a target software system by observing usersbehaviour and deciding whether actions performedare symptomatic of an attack. Misuse-based IDSstry to detect intruders matching the actual user be-haviour with known signatures of malicious behaviour.Anomaly-based IDSs try to detect behaviours that aredifferent from what it is considered to be the normalactivity. Both approaches have been inspired by theinner mechanisms of the human immune system [For-rest et al., 1997; Somayaji et al., 1997], and are widelyexplored in the literature. In this paper, we draw use-ful principles from the human immune system and ap-ply them at the architectural level, while relying onprevious work by IDS community for the algorithmicaspects.

In our case study we consider three main features:

independent layers, distribution and adaptivity. In or-der to achieve better protection we design our securitysystem as made of independent layers:• static non-specific mechanisms—like the skin of

animals—are implemented through authentica-tion;

• static specific mechanisms are typically realisedby authorisation policies;

• dynamical issues are the main concern of anomalyand misuse detection, and resembles the humanacquired immune system.

Responsibilities of our security system should be dis-tributed across several entities—like lymphocytes inacquired immune system—in order to avoid the single-point-of-failure problem. The dynamic layer shouldraise its defences if under attack while releasing com-putational resources when not needed, i.e., adaptively :this point in particular is the main subject of investi-gation of this article. We follow the “self-organisation”approach, i.e., designing a strategy based on sim-ple rules exploiting coupled positive/negative feedbackloops. We investigate stochastic π-calculus modelsof several strategies, and run simulations for severalparameters values using the Stochastic Pi-Machine(SPiM) [Phillips, 2005].

3 Security in MAS

In this section we describe our reference architecturefor MASs based on the TuCSoN coordination infras-tructure [Omicini and Zambonelli, 1999], showing howthis fits an anomaly detection layer.

We consider a system that provides agents withservices encoded in terms of coordination artifacts,i.e., runtime abstractions encapsulating and provid-ing a coordination service, to be exploited by agentsin social contexts expressed by coordination rules andnorms [Omicini et al., 2004]. Following the generalmodel of artifact [Viroli et al., 2005], a coordinationartifact could be characterised by a usage interface, aset of operating instructions, and a coordination be-haviour specification, which can be exploited ratio-nally by cognitive agents.

Accesses of agents to these resources is restricted byan authentication procedure. When an agent entersthe system an authorisation policy limits its actions al-lowing the exploitation of a limited set of services andresources—e.g. those it has payed for. An agent actsupon the environment via a run-time structure, theACC, which enables and rules the interaction betweenthe agent and the environment [Omicini, 2002]— sothe ACC is the right place where to embed authori-sation policies. The whole architecture is depicted inFigure 1.

Even if the two mechanisms of authentication andauthorisation are considered to guarantee a reason-able degree of protection, we promote the idea—aspointed out in the ID community—that a system isbetter protected by additional dynamic mechanisms.Correspondingly, we introduce a layer aimed at de-tecting anomalies in agents behaviour inspired by the

Figure 1: In our reference architecture of MASs, en-vironments are populated by artifacts and agents.

Figure 2: Comparison between the average behaviourof agents and individual one as the basis for anomalydetection: possible actions are grouped into classes.

immune system as well as by previous works on IDSs[Forrest et al., 1997; Somayaji et al., 1997].

We consider agents willing to exploit a specific ar-tifact: an artifact provides a finite set of services, butwe can group them into classes. For the sake of sim-plicity we consider only five classes from A to E. Ifwe trace the agents behaviour “for a while” then weare able to create an average distribution of actionsover that resource: that distribution is taken as the“normal way” for agents to interact with a particu-lar artifact (Figure 2 left). Note that this approach isvalid under two hypotheses:

1. the number of traces is such that the data is sta-tistically significant;

2. the percentage of agents exhibiting abnormal be-haviour is sufficiently low during the initial ob-servation stage.

From now on it is possible to observe an individualagent in order to build its particular distribution ofactions: the deviation from the average distributionmight be a symptom of intrusion or abnormal activ-ity. For instance, if an action belonging to class Cis critical then agent X (Figure 2 right) should beinspected to decide whether it is acting properly ormight cause problems. Note that there exist betterapproaches than the one above described: indeed forus it is just a case to investigate a methodology basedon formal languages and simulation.

Referring to the previous section we describe now

how the security layer fits into the general architec-ture. Basically we need three artifacts, (i) for pro-viding the service, (ii) for logging purpose and (iii)to report abnormal activity. Due to space constraintswe cannot provide the details on how to program thevarious artifacts to achieve the various behaviours: in-terested readers can refer to [Gardelli et al., 2005a] foran overview, or download the whole source code.2

We consider two classes of agents: the ones that re-quest services and the ones that perform inspection.The main role of an inspector is to compare the aver-age signature with the individual ones and report anyanomaly: the inspection for obvious reasons cannotbe performed over all the agents but it will be donethrough sampling. We can consider two parametersrelated to an inspector: (i) the rate of inspection and(ii) the number of inspectors active simultaneously.Note that it is functionally equivalent to have one in-spector working at rate krinsp or k inspectors workingat rate rinsp: as previously stated distributed entitieslet us avoid the single-point-of-failure problem.

In the remainder of this paper we will briefly in-troduce the π-calculus process algebra and show howit can be fruitfully applied to this case so as to pre-dict the global behaviour of the system and choose astrategy for managing the number of inspectors.

4 The π-Calculus

The π-calculus is a formal language developed to rea-son about concurrency [Milner et al., 1992]. Initially,it was intended for describing and analysing systemsconsisting of agents (or processes) which interact witheach other. The basic entity is the name, which is usedas an unstructured reference to a synchronous chan-nel where messages can be sent and received. The se-mantics of π-calculus can be described by a transitionsystem, where the transition relation P −→ P ′ —aprocess P moving to P ′ by the occurrence of an innerinteraction—is defined by operational rules [Milner etal., 1992].

In general, each formal model whose semantics isgiven by a transition system can be extended to astochastic version, resorting to the idea of Markovtransition system. There, each transition P

r−→ P ′

is labelled with a rate r, a non-negative real valuethat describes how the transition probability betweenP and P ′ increases with time.

Stochastic models allow for quantitative simula-tions, for rates can be used to express aspects such asprobability, speed, delays, and so on. One of the no-table stochastic extensions to the π-calculus was intro-duced by Priami [Priami, 1995]. Each channel name isassociated with an activity rate r: the delay of an in-teraction through that channel—representing the useof a resource—is then a random variable with an ex-ponential distribution defined by r. Given a channelname x, the probability pi of a transition P

ri−→ Pi rep-resenting an interaction through x is the ratio between

2The source code for the experiments, π-calculusspecifications and charts can be downloaded fromhttp://www.alice.unibo.it/download/spim/at2ai-5.zip.

its rate ri and the sum of rates of the n transitionsthrough x enabled by P:

pi =ri∑

j=1..n rj, 1 ≤ i ≤ n . (1)

Because of space constraints we are not going to de-scribe here how to write stochastic π-calculus speci-fications since it has already been widely covered bythe literature—see for instance [Milner et al., 1992;Phillips and Cardelli, 2004]. In the next section wediscuss how to exploit stochastic π-calculus in orderto simulate the dynamics of the previously-describedanomaly detection system: we use the Stochastic piMachine (SPiM) [Phillips, 2005] in order to run sim-ulations directly from the specifications.

5 Simulating the Dynamics of anAnomaly Detection System

In this section we focus on the description of modelsfor anomaly detection systems, and on the analysisof simulation results: interested readers can downloadthe specifications used to derive the simulations, alongwith the coloured version of the plots.

The main objective here is to evaluate a numberof different strategies letting inspectors adapt to thechanging number of malicious agents: in other words,we would like the system to self-regulate—raising itsdefences when needed, otherwise releasing computa-tional resources. For the sake of simplicity we considerhere neither missed detections nor false alarms: whilethese are key evaluation parameters for detection al-gorithms, the focus here is on the overall qualitativeMAS behaviour rather than on performances.

In the basic scenario, we consider an agent that per-forms inspections at a fixed rate drawing randomlyan agent from the population: the initial populationof the system is made of 90 normally-behaving agentsand 10 abnormally-behaving ones. While the numberof normal agents remains constant, malicious ones en-ter the system at an arbitrary rate: hence, if the rate ofinspection is 10.0 then statistically the rate of inspec-tion of malicious agents is at least 1.0 and increasesas more malicious agents enter the environment.

Once this model is specified in stochastic π-calculus,the dynamics of the system can be simulated by us-ing SPiM: each simulation adopted a different3 arrivalrate for the malicious agents. From the chart in Fig-ure 3 it is possible to observe—although it was easilypredictable—that an agents inspecting once about ev-ery 10 time units is able to contain malicious agentsonly in case the arrival rate is sufficiently low.

We now extend the basic model in order to changethe number of inspector agents according to the num-ber of malicious agents, which is an unknown from theviewpoint of inspectors. Drawing inspiration from thehuman immune system and the well-known predator-prey system, we introduce the ability of inspectors to

3For the sake of clarity, the charts included show dataonly for a couple of the parameters values in order to high-light the trends.

0

50

100

150

200

250

300

350

0 100 200 300 400 500 600 700 800 900 1000

nu

mb

. o

f a

ge

nts

time

rate 2.0rate 5.0

rate 10.0

Figure 3: The population of malicious agents rapidlygrows by increasing the rate of arrivals. With a sin-gle inspector, the system is able to contain maliciousagents only in the case that the rate of arrivals is com-parable to the rate of inspections.

0

5

10

15

20

25

0 50 100 150 200 250 300 350 400

nu

mb

. o

f a

ge

nts

time

Malicious AgentsInspector Agents

Figure 4: The dynamic equilibrium showed in thissystem is very similar to the one exhibited by a prey-predator-like system. The size of the malicious pop-ulation evolves periodically due to the adaptively-changing inspectors population.

clone themselves. In particular the stimulus for thecloning process is the anomaly-detection event: hence,every time an inspector detects a malicious agent itclones itself, while it dies if it inspects a normal agent.The infrastructure takes care that there is at least aninspector active. The anomaly-detection event trig-gers a positive feedback on the number of inspectors,while the death upon inspection of a normal agent pro-vides the negative feedback necessary to stabilise thepopulation of both inspectors and malicious agents—see chart in Figure 4. Chart in Figure 5 shows thatwhatever the rate of arrivals of malicious agents, thesystem always reaches a dynamic equilibrium. Whilethe strategy described above is somewhat effective, theequilibrium achieved might not be suitable—i.e., theaverage number of malicious agents may still be toohigh.

So, we experiment with a new strategy, which is arefinement of the previous one: we introduce a mem-ory of previous attacks—a mechanism inspired by thehuman immune system. To this end, we add the life-

Figure 5: Increasing the rate of arrivals of maliciousagents the population reach a dynamic equilibriumwhich might however be unsuitable: this is achievedusing inspectors able to clone themselves.

0

5

10

15

20

25

0 50 100 150 200 250 300 350 400

nu

mb

. o

f a

ge

nts

time

rate 0.1rate 10.0

Figure 6: Adding the lifetime feature to inspectors re-sults in a better equilibrium achieved: lifetime of in-spectors is constant while the rate of arrivals increases.Compare this chart with the one in Figure 5.

time property to inspectors: each time an agent per-forms an inspection its lifetime is decreased, but if itdetects a malicious agent then its lifetime is extended.In this sense the lifetime is measured in terms of num-ber of inspections allowed. It is possible to notice fromthe chart in Figure 6 that the equilibrium reached ismuch more reasonable—i.e., the number of maliciousagents is lower—hence the system is able to limit theactivity of malicious agents. In particular, the chart inFigure 7 makes the relation between inspectors’ life-time and the system dynamic equilibrium mostly ev-ident: as the lifetime value increases, the equilibriumstabilises at lower values. Note that, given the targetquality-level, there is trade-off between lifetime valueand use of computational resources.

6 Conclusion and Future Works

One of the general aims of this research line is pavingthe way toward an agent-oriented framework for en-gineering self-organising applications. In [Gardelli etal., 2005a] we initially focused on the applicability ofstochastic π-calculus and the related modelling pro-

0

5

10

15

20

25

30

35

40

45

0 50 100 150 200 250 300 350 400

nu

mb

. o

f a

ge

nts

time

Lifetime 30Lifetime 1

Figure 7: Holding constant the rate of arrivals of ma-licious agents, it is possible to observe that the longerthe inspectors’ lifetime, the better the dynamic equi-librium achieved. Lifetime is measured in terms ofnumber of inspections allowed.

cess. Here, instead, we gave more details on domain-specific issues, i.e., to support anomaly detection forMAS and targeting the simulations to the specificcase. This paper is largely based on [Gardelli et al.,2005b], where we showed how to exploit π-calculusspecifications to quickly investigate the dynamics ofself-organising systems. Here, however, a clear exper-imental evidence has been given to the emergence ofglobal system properties such as the dynamic equi-librium achieved in prey-predator like systems, thusproviding more support for our claims.

According to our reference architecture a MAS ispopulated both by agents and artifacts: the latter em-bed resources or services to be exploited by agents.The system depicted is based on the TuCSoN coor-dination model, and has been prototyped upon theTuCSoN infrastructure. For the architecture and gen-eral principles we took inspiration from the humanimmune system and previous works on IDSs. Forthe methodology, we relied on simulations and mod-elling via stochastic π-calculus, which—even thoughis a quite new language in the context of the MAScommunity—showed its effectiveness as a tool to in-vestigate several design strategies.

While our experiments need to be further detailed,we believe they generally emphasise the ability of theproposed approach to help the MASs developers toanticipate design decisions and strategies at the earlystages of design, before actually developing prototypesand testing them.

References

[Abadi and Gordon, 1997] Martın Abadi and An-drew D. Gordon. A calculus for cryptographic pro-tocols: the spi calculus. In 4th ACM conference onComputer and Communications Security (CCS’97),pages 36–47, New York, NY, USA, 1997. ACMPress.

[Baeten, 1990] Jos C. M. Baeten, editor. Applicationsof Process Algebra, volume 17 of Cambridge Tracts

in Theoretical Computer Science. Cambridge Uni-versity Press, Cambridge, UK, September 1990.

[Forrest et al., 1997] Stephanie Forrest, Steven A.Hofmeyr, and Anil Somayaji. Computer immunol-ogy. Communications of the ACM, 40(10):88–96,1997.

[Gardelli et al., 2005a] Luca Gardelli, Mirko Viroli,and Andrea Omicini. On the role of simulation inthe engineering of self-organising systems: Detect-ing abnormal behaviour in MAS. In Flavio Corra-dini, Flavio De Paoli, Emanuela Merelli, and An-drea Omicini, editors, AI*IA/TABOO Joint Work-shop “Dagli oggetti agli agenti: simulazione e anal-isi formale di sistemi complessi” (WOA 2005),pages 85–90, Camerino, MC, Italy, 14–16 Novem-ber 2005. Pitagora Editrice Bologna.

[Gardelli et al., 2005b] Luca Gardelli, Mirko Viroli,and Andrea Omicini. On the role of simulationsin engineering self-organizing MAS: the case of anintrusion detection system in TuCSoN. In SvenBrueckner, Giovanna Di Marzo Serugendo, DavidHales, and Franco Zambonelli, editors, 3rd Interna-tional Workshop “Engineering Self-Organising Ap-plications” (ESOA 2005), pages 161–175, AAMAS2005, Utrecht, The Netherlands, 26 July 2005.

[Gilmore and Hillston, 1994] Stephen Gilmore andJane Hillston. The PEPA workbench: A tool tosupport a process algebra-based approach to perfor-mance modelling. In Gunter Haring and GabrieleKotsis, editors, Computer Performance Evaluation,number 794 in LNCS, pages 353–368. Springer,1994. Modelling Techniques and Tools. 7th Interna-tional Conference, Vienna, Austria, 3–6 May 1994.Proceedings.

[Heylighen, 2003] Francis Heylighen. The scienceof self-organization and adaptivity. In Knowl-edge Management, Organizational Intelligence andLearning, and Complexity, The Encyclopedia of LifeSupport Systems. EOLSS Publishers, 2003. Avail-able online at http://www.eolss.net.

[Horn, 2001] Paul Horn. Autonomic computing:IBM’s perspective on the state of information tech-nology. Technical report, IBM Corporation, 15 Oc-tober 2001.

[Kephart and Chess, 2003] Jeffrey O. Kephart andDavid M. Chess. The vision of autonomic comput-ing. Computer, 36(1):41–50, January 2003.

[Milner et al., 1992] Robin Milner, Joachim Parrow,and David Walker. A calculus of mobile processes,part I. Information and Computation, 100(1):1–40,September 1992.

[Omicini and Denti, 2001] Andrea Omicini and En-rico Denti. From tuple spaces to tuple centres.Science of Computer Programming, 41(3):277–294,November 2001.

[Omicini and Zambonelli, 1999] Andrea Omicini andFranco Zambonelli. Coordination for internet ap-plication development. Autonomous Agents andMulti-Agent Systems, 2(3):251–269, 1999.

[Omicini et al., 2004] Andrea Omicini, AlessandroRicci, Mirko Viroli, Cristiano Castelfranchi,and Luca Tummolini. Coordination artifacts:Environment-based coordination for intelligentagents. In Nicholas R. Jennings, Carles Sierra, LizSonenberg, and Milind Tambe, editors, 3rd Inter-national Joint Conference on Autonomous Agentsand Multiagent Systems (AAMAS 2004), volume 1,pages 286–293, New York, USA, 19–23 July 2004.ACM.

[Omicini, 2002] Andrea Omicini. Towards a notion ofagent coordination context. In Dan C. Marinescuand Craig Lee, editors, Process Coordination andUbiquitous Computing, chapter 12, pages 187–200.CRC Press, October 2002.

[Phillips and Cardelli, 2004] Andrew Phillips andLuca Cardelli. Simulating biological systems in thestochastic pi-calculus. Technical report, MicrosoftResearch, Cambridge, UK, July 2004.

[Phillips, 2005] Andrew Phillips. The Stochastic PiMachine (SPiM), 2005. Version 0.041 available on-line at http://www.doc.ic.ac.uk/˜anp/spim/.

[Priami, 1995] Corrado Priami. Stochastic pi-calculus. Computer Journal, 38(7):578–589, 1995.

[Regev et al., 2001] Aviv Regev, William Silverman,and Ehud Shapiro. Representation and simulationof biochemical processes using the pi-calculus pro-cess algebra. In Russ B. Altman, A. Keith Dunker,Lawrence Hunter, and Teri E. Klein, editors, 6thPacific Symposium on Biocomputing (PSB 2001),volume 6, pages 459–470, Hawaii, USA, 3-7 Jan-uary 2001. World Scientific Press.

[Somayaji et al., 1997] Anil Somayaji, StevenHofmeyr, and Stephanie Forrest. Principles ofa computer immune system. In Tom Haigh,Bob Blakley, Mary Ellen Zurbo, and CatherineMeodaws, editors, 1997 Workshop on New SecurityParadigms (NSPW’97), pages 75–82, New York,NY, USA, 23–26 September 1997. ACM Press.

[Tofts, 1992] Chris Tofts. Describing social insect be-haviour using process algebra. Transactions of theSociety for Computer Simulation, 9:227–283, De-cember 1992.

[Viroli et al., 2005] Mirko Viroli, Andrea Omicini,and Alessandro Ricci. Engineering MAS environ-ment with artifacts. In Danny Weyns, H. Van DykeParunak, and Fabien Michel, editors, 2nd Inter-national Workshop “Environments for Multi-AgentSystems” (E4MAS 2005), pages 62–77, AAMAS2005, Utrecht, The Netherlands, 26 July 2005.