dash locality hierarchies (padal'16)

Tobias Fuchs

[email protected]

Ludwig-Maximilians Universität (LMU) München

DASH Locality Hierarchies

An Information System for Hierarchical Locality

Motivation

Motivation:Portable efficiency of ported applications like LULESH and graph applications for heterogeneoussystems

hwloc static hierarchical hardware topology

DASH locality as view from variable run-time configuration

… extended to full distributed topology… supporting heterogeneous systems, esp. Intel MIC… designed for use cases like hierarchical graph partitioning


Trees vs. Reality

We do not model hardware locality as trees

… because many represented systems are anything but trees


http://frankdenneman.nl/2016/07/11/numa-deep-dive-part-3-cache-coherency/

Related Concepts

Combining existing abstractions

- Hierarchical Place Treesabstraction of locality scopes and process mapping

- hwlocnode-level machine topology and hardware capacityfocuses at topology discovery (once, at startup) and provides queries on topology data

Locality Hierarchies

- are specifically designed for user-specified views and modifications of the topology representation

- are specifically designed for hierarchical process structures (teams)


Examples: Xeon PHI


Compute Node with Xeon Phi Accelerators as seen by hwloc

Examples: Xeon PHI

Compute Node with Xeon Phi Accelerators as seen by DASH


Examples: Xeon PHI


Key Functionalities

Why we need locality hierarchies in DART/DASH:

- Locality-optimized grouping of processes into teams

- Load-balancing, requires: topology information

… obviously, to find suitable processes for balancing hardware capacities

… like number of cores/threads and (shared) memory capacitiesavailable to processes

- Dynamic distance measures: variable at run time instead of staticdistance matrix


Locality Domains


Splitting / Grouping


Heterogeneous min_element


Some fundamental operations on locality hierarchies

Basic principle of usage:( (filter/select) (group/split) )*

C API, bindings for Python in development, Fortran bindings feasible

C API


Usage of C++ API

// split into num_groups teams at NUMA locality scopeauto & new_team = dash::Team::All().locality_split(

dash::util::Locality::Scope::NUMA,// optional, defaults to one team per locality scopenum_groups);

// split into two teams: leader team and workersauto & new_team = dash::Team::All().leader_split(

dash::util::Locality::Scope::NODE);

// split into teams by predicateauto & new_team = dash::Team::All().specific_split(

[](dash::util::LocalityDomain ld) {// ...return new_team_id_for_ld;

});


Leader Groups


Logical vs. physicaltopology

Domain Aliasing


Splitting / Grouping


Basic Distance Model


Links


www.dash-project.org

www.github.com/dash-project

Stand-alone library of DASH locality hierarchies (libdhloc)available in ~4 weeks

http://www.dash-project.org/

http://www.github.com/dash-project

dash locality hierarchies (padal'16)

Science