illumosThe State of Fully Modern Unixbrian.bennett@joyent.com@bahamat

illumos in a Nutshell• Is an open source descendent of OpenSolaris

• …which itself was a branch of Solaris Nevada

• …which was the name of the release after Solaris 10

• …and was open but is now closed

• …and is itself a descendent of Solaris 2.x

• …but it can all be called “SunOS 5.x”

• …but not “SunOS 4.x” — that’s different

• …which came from Unix

Unix Family Tree

Unix Family Tree

Unix: Grandfather of Open Source

• At the dawn of computing software was seen as having no commercial value

• Computers came with operating systems and operating systems came with source code

• Prohibited from entering the computing industry by the 1956 consent decree, AT&T initially could not commercialize Unix

• Sixth Edition was widely circulated among universities

• The culture of freely shared source code is what made Unix thrive

• Graduating from universities to the enterprise, by 1983 Unix was available from about 60 different vendors

The Proprietary Middle Ages

• The breakup of AT&T in 1982 changed things for Unix, with the consent decree lifted, Bell Labs began to strongly commercialize it

• The GNU Manifesto and Free Software Foundation were created in direct opposition to this proprietary environment

• In 1991, at the height of the Unix Wars, Linux was first made available under the GNU GPL v2 license

• The 1992 USL v. BSDi lawsuit cast a cloud of uncertainty around BSD code

• Nature abhorring a vacuum, Linux inherited the spirit of freedom classically enjoyed by Unix users

• Meanwhile hardware vendors all but abandoned Unix, mobilizing a strategy around Windows NT, all except Sun

Solaris

• The rise of Linux and x86 micros forced the market price of OS acquisition to zero, it became clear that open sourcing Solaris was the right business decision for Sun

• Solaris had always been a loss-leader for Sun; the company made money from complementary goods (hardware, support, etc.)

• Moreover, open sourcing the system would assure that Solaris innovations would transcend products and become advances in the state of the art

• As early as the late ’90s, there was serious talk within Sun of open sourcing Solaris

• In 2004 Jonathan Schwartz, then CEO of Sun, made it a priority

OpenSolaris

• In January 2005, OpenSolaris was born when Dtrace became the first of the system to be open sourced, from its very first release

• Unfortunately, not all elements of Solaris could be open sourced; some contracts prevented some small (but important) bits from being open sourced

• To allow for such proprietary drivers, Sun developed a file-based copy-left license, the Common Development and Distribution License (CDDL)

• The rest of the OS was open sourced in June 2005 and was developed in the open from that point — changes to the OS were open sourced as they integrated

The Death of OpenSolaris

• Ailing Sun was bought by Oracle in 2009, with the acquisition closing in February 2010

• It became clear that Oracle had absolutely no interest in OpenSolaris

• Despite that it would be obviously stupid, there was even a move within Oracle to close the system

• Throughout 2010 there was a mass exodus of Solaris talent away from Oracle

The Birth of illumos

• Starting in the summer of 2010, Garrett D’Amore at Nexenta — with help from Rich Lowe, Jason King and others — began the process of either writing the closed bits from scratch or porting them from BSD

• By early August, an entirely open system was booting

• Dubbed “illumos” (from illuminare, Latin for illuminate) and made available August 3, 2010

• Notably, the announcement included both code and a working demo — leading with technology, not rhetoric

• illumos was not intended to be a fork, but rather an entirely open downstream repository of OpenSolaris

The Silent Death of OpenSolaris

• On Friday, August 13th, 2010 an internal memo was circulated by the putative Solaris leadership:

We will distribute updates to approved CDDL or other open source-licensed code following full releases of our enterprise Solaris operating system. In this manner, new technology innovations will show up in our releases before anywhere else. We will no longer distribute source code for the entirety of the Solaris operating system in real-time while it is developed, on a nightly basis.

• This was never publicly announced; updates just silently stopped on August 18th, 2010.

• And it was a lie anyway: Solaris 11 was released on November 9, 2011 — and there was no source release

The Phoenix

• illumos is what Sun intended it to be, a fully open and free operating system

• Oracle (silently) forked Solaris by leaving the community

• Oracle's abandonment of the community only strengthens illumos’ position as the repository of record for Solaris technologies and the future of the platform

• By refusing to take part in the community, Oracle is denying themselves features developed in the community

• Today illumos embodies the spirit of, and is the proper successor to Solaris

illumos Differentiators

• Dtrace & MDB

• ZFS

• Zones

• LX-brand

• Fault Management Architecture & Service Management Facility

• Crossbow

• Role Based Access Control & Least Privilege

• KVM

DTrace & MDB

• DTrace is a magician that conjures up rainbows, ponies and unicorns — and does it all entirely safely and in production!

• Instruments all software, applications, databases, web servers, the operating system kernel and device drivers

• Allows you to ask arbitrary questions about what the system is doing and find the answer

• Only as hard as you think it is — D syntax inspired by awk and C

• MDB, the modular debugger

• Trivially identify the cause of crashes

ZFS

• 128-bit, copy on write filesystem

• Pooled Storage

• End to end data integrity

• Self healing, with sufficient replicas

• Transactional writes, always consistent on disk (no fsck!)

• Snapshots & clones, send & receive

• Lightweight filesystem creation with arbitrary mount points

• Block oriented zvols

Zones

• Inspired by BSD Jails, aped by Linux

• Takes the concept of jails to its fullest logical conclusion, an entire virtual operating system

• Zones provide strong isolation, no process running in a non-global zone, even one with superuser credentials is allowed to view or affect activity in other zones

• No virtualization overhead — apps run at full speed, executing on bare metal

• In 2014, encouraged by initial work from illumos community member David Mackay, Joyent reintroduced LX-brand zones, allowing Linux binaries to run unmodified on SmartOS

Predictive Self-Healing

• Fault Manager (FMA)

• Errors are detected and sent to the Fault Manager (fmd)

• Fmd dispatches error reports to the diagnosis engine

• Faults are automatically corrected, or actionable reports raised to administrators

• Service Manger Facility (SMF)

• Milestones and Service dependencies

• Services identified by Fault Management Resource Identifier (FMRI)

• Faulted (crashed) services are automatically restarted

• Unhealthy services are raised to administrators

Crossbow: Network Virtualization

• Physical interfaces, VNICs, VLANs, etherstubs (virtual switch), link aggregation, 802.1D bridges, IP tunneling, network overlay (VxLAN)

• Physical interfaces are also etherstubs for attaching VNICs

• Interface independent etherstubs can be created, to which VNICs can be attached

• VNICs can be shared or exclusive IP

• Restrict

• MAC spoofing

• IP spoofing

• DHCP spoofing

• Non-IP traffic

Role Based Access Control

• Fine grained privilege control

• Profiles — a named set of privileges granted to a role or user

• Roles — similar to users, but can’t log in directly; assume a role to gain privileges

• E.g., a profile allowing administration of the Apache SMF service would not allow a user to run httpd and bind to port 80.

KVM

• Originally ported from Linux in 2010

• See, GPL & CDDL code can live together!

• Run foreign operating systems in a Virtual Machine

• Linux

• FreeBSD

• Windows

• Etc.

• KVM instances run inside a special KVM-brand zone

illumos Distributions• NexentaStor — From Nexenta. Commercial distribution aimed at storage appliances.

Closest spiritual successor to Fishworks/Sun Storage.

• OmniOS — From OmniTI. A minimalist base OS geared towards server class systems. Closest spiritual successor to Solaris 10, path of least resistance to move from closed to open Solaris.

• OpenIndiana — Derived from the original Sun OpenSolaris reference distribution, suitable for use on servers and desktops. Aims to be binary compatible with Solaris 11.

• OpenSXCE — An up-to-date distribution in sync for SPARC and x86. Supports everything between Ultra-1 (1995) to everything that came to market before 2011.

• SmartOS — From Joyent. Designed for cloud computing. It includes a read-only platform image, port of KVM, and LX-brand zones.

• Tribblix — SVR4 packaging, providing a retro feel with modern components

illumos Distributions• NexentaStor — From Nexenta. Commercial distribution aimed at storage appliances.

Closest spiritual successor to Fishworks/Sun Storage.

• OmniOS — From OmniTI. A minimalist base OS geared towards server class systems. Closest spiritual successor to Solaris 10, path of least resistance to move from closed to open Solaris.

• OpenIndiana — Derived from the original Sun OpenSolaris reference distribution, suitable for use on servers and desktops. Aims to be binary compatible with Solaris 11.

• OpenSXCE — An up-to-date distribution in sync for SPARC and x86. Supports everything between Ultra-1 (1995) to everything that came to market before 2011.

• SmartOS — From Joyent. Designed for cloud computing. It includes a read-only platform image, port of KVM, and LX-brand zones.

• Tribblix — SVR4 packaging, providing a retro feel with modern components

SmartOS

• Read-only platform image

• Boot from USB or PXE

• Upgrade? Downgrade? Just boot to the desired platform image

• Global zone is for zone management

• Everything else runs in a zone

• KVM in a reduced privilege zone

• SmartOS zones use pkgsrc (pkgin) from NetBSD for packages (>14,000 packages, on par with Debian)

SmartDataCenter

• Orchestration stack for a datacenter of SmartOS Compute Nodes

• Up and running in about 30 minutes

• Robust API and Web UI

• Open source! MPLv2 license

• All the code is on github

• github.com/joyent/sdc

• Joyent Public Cloud is entirely managed with SmartDataCenter

• Download and run it yourself

• https://us-east.manta.joyent.com/Joyent_Dev/SmartDataCenter

Docker Revolution

• Docker has used the rapid provisioning + shared underlying filesystem of containers to allow developers to think operationally

• Developers can encode dependencies and deployment practices into an image

• Images can be layered, allowing for swift development

• Images can be quickly deployed — and redeployed

• Docker will do to apt what apt did to tar

Docker Frustrations

• Linux container runtime vulnerabilities require running containers in VMs or on single tenant hardware

• Managing VM or hardware life cycles adds complexity to deployments

• Running containers in VMs erases the performance advantages of containerized OS virtualization

• Network implementation is host-centric, making it difficult to connect containers on different hosts

• Managing multiple Docker hosts increases complexity

Joyent Triton

• Run Docker images natively in SmartOS zones

• No VM, no overhead — the way God intended!

• Datacenter presented as a single Docker host, no hosts to manage

• Each container gets its own exclusive IP stack, with optional global IP

• Resource isolation, hardened security context, dedicated networking all solved by Zones 10 years ago

• Native SmartOS debugging (dtrace, mdb) along side Linux native binaries

• In JPC, LX-brand GA since April, Docker GA coming soon

• Or today in your own datacenter

Beta

Thank You!

• http://www.illumos.org/

• http://www.joyent.com/

• http://www.smartos.org/

• http://www.omniti.com/

• http://www.openindiana.org/

• irc.freenode.net

• #illumos

• #smartos

• #omnios