dash locality hierarchies (padal'16)

Tobias Fuchs

fuchst@nm.ifi.lmu.de

Ludwig-Maximilians Universität (LMU) München

DASH Locality Hierarchies

An Information System for Hierarchical Locality

Motivation

Motivation:Portable efficiency of ported applications like LULESH and graph applications for heterogeneoussystems

hwloc static hierarchical hardware topology

DASH locality as view from variable run-time configuration

… extended to full distributed topology… supporting heterogeneous systems, esp. Intel MIC… designed for use cases like hierarchical graph partitioning

DASH Locality Hierarchies

Trees vs. Reality

We do not model hardware locality as trees

… because many represented systems are anything but trees

DASH Locality Hierarchies

http://frankdenneman.nl/2016/07/11/numa-deep-dive-part-3-cache-coherency/

Related Concepts

Combining existing abstractions

- Hierarchical Place Treesabstraction of locality scopes and process mapping

- hwlocnode-level machine topology and hardware capacityfocuses at topology discovery (once, at startup) and provides queries on topology data

Locality Hierarchies

- are specifically designed for user-specified views and modifications of the topology representation

- are specifically designed for hierarchical process structures (teams)

DASH Locality Hierarchies

Examples: Xeon PHI

DASH Locality Hierarchies

Compute Node with Xeon Phi Accelerators as seen by hwloc

Examples: Xeon PHI

Compute Node with Xeon Phi Accelerators as seen by DASH

DASH Locality Hierarchies

Examples: Xeon PHI

DASH Locality Hierarchies

Examples: Xeon PHI

DASH Locality Hierarchies

Key Functionalities

Why we need locality hierarchies in DART/DASH:

- Locality-optimized grouping of processes into teams

- Load-balancing, requires: topology information

… obviously, to find suitable processes for balancing hardware capacities

… like number of cores/threads and (shared) memory capacitiesavailable to processes

- Dynamic distance measures: variable at run time instead of staticdistance matrix

DASH Locality Hierarchies

Locality Domains

DASH Locality Hierarchies

Splitting / Grouping

DASH Locality Hierarchies

Heterogeneous min_element

DASH Locality Hierarchies

Some fundamental operations on locality hierarchies

Basic principle of usage:( (filter/select) (group/split) )*

C API, bindings for Python in development, Fortran bindings feasible

C API

DASH Locality Hierarchies

Usage of C++ API

// split into num_groups teams at NUMA locality scopeauto & new_team = dash::Team::All().locality_split(

dash::util::Locality::Scope::NUMA,// optional, defaults to one team per locality scopenum_groups);

// split into two teams: leader team and workersauto & new_team = dash::Team::All().leader_split(

dash::util::Locality::Scope::NODE);

// split into teams by predicateauto & new_team = dash::Team::All().specific_split(

[](dash::util::LocalityDomain ld) {// ...return new_team_id_for_ld;

});

DASH Locality Hierarchies

Leader Groups

DASH Locality Hierarchies

Logical vs. physicaltopology

Domain Aliasing

DASH Locality Hierarchies

Splitting / Grouping

DASH Locality Hierarchies

Splitting / Grouping

DASH Locality Hierarchies

Basic Distance Model

DASH Locality Hierarchies

Links

DASH Locality Hierarchies

www.dash-project.org

www.github.com/dash-project

Stand-alone library of DASH locality hierarchies (libdhloc)available in ~4 weeks