AliExpress’Way To Microservices

Juven Xu (许晓斌) Senior Tech Expert at AliExpress

About Me

Juven has 10 years' experience on software development.

He's currently leading a team in AliExpress, focusing on improve technical productivity and stability using methods like Microservices and DevOps.

He authored a book about Apache Maven.

Twitter: @juvenxu

A subsidiary of Alibaba Group.

The largest cross border trading platform in the world.

In 2016 11.11 shopping festival, from 230 countries, over 6 millions consumers placed 35.78 millions of orders.

Agenda

AliExpress’ Technical EchosystemA

PrinciplesB

PracticesC

Back in 2013 …

release queue

dependency hell

unable to start locally

we had only a dozen of applications but more than one hundred of developers, so developers have to create a branch, and merge it before releasing, and usually have to wait for other people, who is releasing the same application.

Too many logics were put into one application, including a lot of very old java libraries, it’s very easy to have dependency collisions (log is gone, runtime issue etc.) It’s quite common to see an developer spend half day resolving a dependency collision issue.

Applications are usually quite complex, running on a customized version of JBoss, needs a few shell scripts to configure and start, it usually takes more than 10 minutes. It’s almost impossible to developers to start his application on his laptop.

Back in 2013 …

svn

maven 2

java 6

JBoss 4

war

SpringFramework 2.x

Servlet API 2.x

Now

Git & Gitlab

Maven 3

Java 8

Embedded Tomcat 8

jar

Spring Boot & Spring Cloud

SpringFramework 4.x

Servlet API 3.1

svn

maven 2

java 6

JBoss 4

war

SpringFramework 2.x

Servlet API 2.x

The Big Picture

DB HBase NoSql

Tair

Service

Service Service Service

Service Service

Web App Web App Web App

HSF

MQ

DB HBase NoSql

Tair

Service

Service Service Service

Service Service

Web App Web App Web App

HSF

MQ

The Big Picture

• AliExpress has hundreds of applicationsmost of which are Java applications.

• Mainly communicated in sync way using HSF (similar to gRPC)

• Some communicated in async way using MetaQ (similar to Kafka)

• Multiple data centers.

• All apps use the same release system and monitoring system.

PrinciplesB

1. Be able to start locally

2. API first

3. Embrace OSS

4. Developers take full responsibility

Principle 1: Be Able To Start Locally

Principle 1: Be Able To Start Locally

• Feedback time: 10+ mins -> 3- mins

• Much easier to understand

• Much easier to debug

The developers heavily rely on our testing environment to deploy their application and do some simple testing. Even for one single line of code! and sometimes the testing environment is not stable enough.

To be able to start up locally, we use SpringBoot, and did a lot of technical debt clean up.* no more jboss* no magic scripts etc.

Principle 2: API First

• The cost of an incompatible change to your API is very high!

• In Java world, don’t pollute your users with unnecessary dependencies.

because we are almost 100% Java based, so people publish their API as client.jar, it’s so easy to put logics into the client.jar, and it quickly become very very fat

1. too much logic in the fat jar2. Too many transitive dependence in the fat jar

image you depends on 10 services, they each give you a fat jar, each fat jar has 40 transitive dependencies…. it’s a nightmare

Principle 3: Embrace OSS

MicroservicesSpringBoot

• Choose the mainstream OSS tools/frameworks, so • More documents/books • The developers have more passion to learn • Can be Googled for problems.

Our R&D teams are getting more and more international, we have developers don’t know Chinese at all. So asking them to learn Alibaba frameworks (usually only has Chinese documents, even comments are Chinese) is almost impossible.

Principle 4: Developers Take Full Responsibility

• Developers are the owner of the applications. • Responsible to the features, reliability and

performance etc.

• This principle is the core of our DevOps culture.

• Comprehensive infrastructure is required • must be simple to use

this is the only scalable way, we have hundreds of applications but only a few ops guys.

the developers have a full understanding about how his applications are running in production

Infrastructure: release system, monitor system

PracticesC

1. Organize codebase based on applications

2. Use Docker to make environment consistent

3. Integrate Alibaba services into Spring Boot

4. Use Spring Cloud Config & Diamond to manage config

5. Use Maven BOM to manage Java depdencies

6. Apply strict rules on published jar

7. Apply strict naming standard on applictions

8. Split microservices from monolithic

1. Organize codebase based on apps

monolithic monolithic monolithic

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

global jar

Yes, code is reused, but …

Global jar: each jar is an svn trunk, and will be deployed into maven repository.

So when we want to release, we usually create a few branches from different global jars and one monolithic, which easily causes collision and release queue.

microservice

local jar local jar

local jar local jar

published jar

microservice

local jar local jar

local jar local jar

published jar

microservice

local jar local jar

local jar local jar

published jar

microservice

local jar local jar

local jar local jar

published jar

local jar and the microservice are in the same git repo, actually local jar only exists in maven build and don’t get into maven repo

published jar is the API of an microservice, it’s semantic versioned, and get into maven repo.

2. User Docker to make environment consistent

FROM reg.docker.alibaba-inc.com/aone-base/ae-boot-app:latest COPY ${APP_NAME}.tgz /home/admin/${APP_NAME}.tgz

Alibaba Base Image

AliExpress Base Image

AliExpress Boot Image

FROM

FROM

Boot App

Boot App

Boot App

Boot App

Boot AppFROM

1. the developers always FROM our latest image, so it’s easy to force upgrade globally (e.g. for security issues)

2. As long as the boot images are well tested, most of the issue about environment are gone. (thanks to immutability)

3. we can speed up the release process by preloading docker app images

3. Integrate Alibaba services into Spring Boot

<dependency> <groupId>com.aliexpress.boot</groupId> <artifactId>spring-boot-starter-hsf</artifactId> </dependency>

@SpringBootApplication @EnableHSF public class DemoApplication{}

@HSFProvider(serviceInterface = UserService.class, serviceVersion = "1.0.1") public class UserServiceImpl implements UserService{}

Use starter to simply code

{ "status": "UP", "geoipServiceReadyIndicator": { "status": "UP" }, "discoveryComposite": { "description": "Spring Cloud Eureka Discovery Client", "status": "UP", "eureka": { "description": "Remote status from Eureka server", "status": "UNKNOWN", "applications": {} } }, "diamond": { "status": "UP", "dataIds": [ "com.aliexpress:application.properties" ] }, "hsf": { "status": "UP" }, "diskSpace": { "status": "UP", }, "refreshScope": { "status": "UP" }, "hystrix": { "status": "UP" } }

http://localhost:7002/health

Comprehensive Health Check

"metrics": { "metaq.test-topic:*.sentPerSecond": { "count": 1, "fifteenMinuteRate": 2.283500961617208E-5, "fiveMinuteRate": 2.9767585188932503E-13, "meanRate": 1.222495102777674E-4, "oneMinuteRate": 1.4608244973590434E-60 }, "metaq.test-topic:*.totalSentFailures": { "count": 0 }, "metaq.test-topic:*.totalConsumed": { "count": 1 }, "metaq.test-topic:*.totalSent": { "count": 1 }, "metaq.test-topic:*.totalConsumedFailures": { "count": 0 }, "metaq.test-topic:*.consumedFailuresPerSecond": { "count": 0, "fifteenMinuteRate": 0.0, "fiveMinuteRate": 0.0, "meanRate": 0.0, "oneMinuteRate": 0.0 }, "metaq.test-topic:*.consumedPerSecond": { "count": 1, "fifteenMinuteRate": 1.2580854264500415E-7, "fiveMinuteRate": 4.8388261531308695E-15, "meanRate": 1.2195116231168655E-4, "oneMinuteRate": 1.0746217926192886E-61 }, "metaq.test-topic:*.sentFailuresPerSecond": { "count": 0, "fifteenMinuteRate": 0.0, "fiveMinuteRate": 0.0, "meanRate": 0.0, "oneMinuteRate": 0.0 } }

metrics using DropWizard metrics

http://localhost:7002/metaq

4. Use Spring Cloud Config and Diamond to manage config

Binarybuild config

Binary X

Binary Y

Binary ZGlobalConfig Repo

Binarybuild

Config Repo

Config Repo

Config Repo

Binary

Binary

Binary

immutable

diamond-spring-boot-starterSpring Boot starter for diamond

<dependency> <groupId>com.alibaba.boot</groupId> <artifactId>diamond-spring-boot-starter</artifactId> </dependency>

1. Configure application.properties :

spring.application.name=archimedes-master spring.application.group=com.aliexpress.archimedes

2. Configure Diamond:com.aliexpress.archimedes:archimedes-master.properties

archimedes.masterEnabled=true eureka.client.fetch-registry=true … …

3. In java, use the configuration like this:

@Value("${archimedes.masterEnabled}") private boolean masterEnabled;

@Value("${archimedes.messagingProducerDestination}") private String messagingProducerDestination;

5. Use Maven BOM to manage Java dependencies

<properties> <alibaba-spring-boot.version>1.4.0</alibaba-spring-boot.version> <spring-boot.version>1.4.2.RELEASE</spring-boot.version> </properties>

<dependencyManagement> <dependencies> <dependency> <groupId>com.alibaba.boot</groupId> <artifactId>alibaba-spring-boot-dependencies</artifactId> <version>${alibaba-spring-boot.version}</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-dependencies</artifactId> <version>${spring-boot.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement>

<dependencies> <dependency> <groupId>com.aliexpress.boot</groupId> <artifactId>spring-boot-starter-hsf</artifactId> </dependency> <dependency> <groupId>com.alibaba.boot</groupId> <artifactId>diamond-spring-boot-starter</artifactId> </dependency> </dependencies>

The BOM defines a set of dependencies people usually use, so when they import it,

1.they don’t need to care about the specific version of a dependency,

2.and don’t need to care about dependency collision.

<properties> <alimonitor-spring-boot-starter.version>1.0.4</alimonitor-spring-boot-starter.version> <diamond-spring-boot-starter.version>1.1.0</diamond-spring-boot-starter.version> <maven-spring-boot-starter.version>1.1.0</maven-spring-boot-starter.version> <spring-boot-starter-hsf.version>1.2.4</spring-boot-starter-hsf.version> <eagleeye-spring-boot-starter.version>1.0.2</eagleeye-spring-boot-starter.version> <spring-boot-starter-endpoints.version>1.0.3</spring-boot-starter-endpoints.version> <region-spring-boot-starter.version>1.0.1</region-spring-boot-starter.version> <jingwei-spring-boot-starter.version>1.1.8</jingwei-spring-boot-starter.version> <schedulerx-spring-boot-starter.version>1.0.0</schedulerx-spring-boot-starter.version> <spring-cloud-stream-starter-metaq.version>1.0.0</spring-cloud-stream-starter-metaq.version> <docker-spring-boot-starter.version>1.0.1</docker-spring-boot-starter.version> <tddl-spring-boot-starter.version>5.1.25</tddl-spring-boot-starter.version> <tair-spring-boot-starter.version>2.2.4</tair-spring-boot-starter.version> <spring-boot-starter-mybatis.version>1.1.0</spring-boot-starter-mybatis.version> <unicorn-spring-boot-starter.version>1.0.0</unicorn-spring-boot-starter.version> </properties>

<dependencyManagement> <dependencies> <dependency> <groupId>com.aliexpress.boot</groupId> <artifactId>spring-boot-starter-endpoints</artifactId> <version>${spring-boot-starter-endpoints.version}</version> </dependency> <dependency> <groupId>com.alibaba.boot</groupId> <artifactId>maven-spring-boot-starter</artifactId> <version>${maven-spring-boot-starter.version}</version> </dependency> <dependency> <groupId>com.aliexpress.boot</groupId> <artifactId>spring-boot-starter-hsf</artifactId> <version>${spring-boot-starter-hsf.version}</version> </dependency> <dependency> <groupId>com.alibaba.boot</groupId> <artifactId>alimonitor-spring-boot-starter</artifactId> <version>${alimonitor-spring-boot-starter.version}</version> </dependency> <dependency> <groupId>com.alibaba.boot</groupId> <artifactId>eagleeye-spring-boot-starter</artifactId> <version>${eagleeye-spring-boot-starter.version}</version>

6. Apply strict rules on published jar

microservice

local jar local jar

local jar local jar

published jarPublished jar is the API of a microservice, usually depended by other microserivces,

it MUST be clean!

microservice

local jar local jar

local jar local jar

published jar

1. do not depend on log impl (log4j, logback etc.) 2. do not have log config (log4j.xml, logback.xml) 3. do not depend on legacy jars 4. do not depend on SNPAHOTs 5. …

<parent> <groupId>com.aliexpress</groupId> <artifactId>aliexpress-published-lib-parent</artifactId> <version>3-SNAPSHOT</version> </parent>

<groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>1.2</version> <executions> <execution> <id>enforce-banned-dependencies</id> <goals> <goal>enforce</goal> </goals> <configuration> <rules> <bannedDependencies> <searchTransitive>false</searchTransitive> <excludes> <exclude>com.alibaba.external:*:*:*:compile</exclude> <exclude>log4j:log4j:*:*:compile</exclude> <exclude>ch.qos.logback:*:*:*:compile</exclude> </excludes> </bannedDependencies> </rules> </configuration> </execution> <execution> <id>enforce-no-log-config</id> <goals> <goal>enforce</goal> </goals> <phase>prepare-package</phase> <configuration> <rules> <requireFilesDontExist> <files> <file>${project.build.outputDirectory}/log4j.properties</file> <file>${project.build.outputDirectory}/log4j.xml</file> <file>${project.build.outputDirectory}/logback.xml</file> </files> </requireFilesDontExist> </rules> <fail>true</fail> </configuration> </execution> </executions> </plugin>

7. Apply strict naming standard on applications

[ae]-[product]-[domain/scenario]-[classifier]

Classifier

1 s - Service, this application expose HSF service 2 f - Front-end Web Application, expose 80 port to internet. 3 b - Back-end Web Application, expose 80 port to intranet 4 d - Data Flow Application, consumes messages, producer messages, or writes data to DB.

Product

An enumerable project name, we are very strict on adding new product name.

e.gae-fileserver2-sync-sae-fileserver2-upload-fae-fileserver2-admin-b

we have hundreds of applications, it’s important to name them in a consistent way so it’s easy to communicate.

all the systems use this name:monitoring system, release system

and we can control the problem domain

8. Split microservices from monolithic

• Developers can learn step by step. • Much lower risk, easy to switch traffic back. • Heavily rely on the monitoring system.

@juvenxujuven.xuxb@alibaba-inc.com

Q & A