building codealike: a journey into the developers analytics world

RavenDB Conference 2016

Building CodealikeA journey into the developers analytics world

Federico Loisfederico.lois@corvalius.comgithub.com/redknightloistwitter.com/federicolois

What do we do?

• We track how developers write code.• How interruptions impact developer’s productivity• What part of the software they worked on.

• An instant (available) extended memory.• How much they can concentrate.• How they compare against other developers.

• In the world,• In their team,• With themselves (a few months/years ago).

What do I mean?

Lets so some actual data!!!

What data is stored?

Time

ActivityArtifact

Activity

System

Outside the

IDE

Debugging

Building

Coding

Artifacts

Time

+ 200,000 Hs.of detailed activity+130

countries + 100M de actividades

+ 8M de artefactos

Duplicating (roughly)

each 6 months

PersistentNon

persistentDERIVED

INFORMATION

Volume

We ingest more data than our ability of “instantly” index it.

Timezones

Aggregation

Months

Weeks

Days

Hours

Minutes

Seconds

Aggregation

Visualization

Aggregation

Months

Weeks

Days

Hours

Minutes

Seconds

Aggregation

Visualization

Evil Indexes

public class WebAggregateByHostIndex : AbstractIndexCreationTask<WebActivityCollection, WebActivityAggregateByUser> { public WebAggregateByHostIndex() {

this.MaxIndexOutputsPerDocument = 2048;

Map = activities => from activity in activities let statesMapping = from state in activity.Activities

let host = state.Url != null ? new Uri(state.Url.ToString()).Host : "unknown" select new

{

Owner = "membership/account/" + activity.Owner,

Status = state.Status, Host = host,

StartTicks = state.StartTimeTicks, DurationTicks = state.DurationTicks,

EndTicks = state.StartTimeTicks + state.DurationTicks, }

from state in statesMapping

group state by new {

Owner = state.Owner, Host = state.Host, } into g

select new WebActivityAggregateByUser {

Owner = g.Key.Owner, Host = g.Key.Host,

MinTimeTicks = g.Min( x => x.StartTicks),

MaxTimeTicks = g.Max( x => x.EndTicks),

Navigation = g.Where(x => x.Status == "navigation").Sum(x => x.DurationTicks), Debugging = g.Where(x => x.Status == "debugging").Sum(x => x.DurationTicks), Debugger = g.Where(x => x.Status == "debugger").Sum(x => x.DurationTicks),

NavigationCount = g.Where(x => x.Status == "navigation").Count(),

DebuggingCount = g.Where(x => x.Status == "debugging").Count(), DebuggerCount = g.Where(x => x.Status == "debugger").Count(),

};

Reduce = results => from result in results group result by new {

result.Owner,

result.Host } into g select new WebActivityAggregateByUser

{ // This is the bag used on the reduce. Owner = g.Key.Owner, Host = g.Key.Host,

// Spatial index to prune the search space.

MinTimeTicks = g.Min(x => x.MinTimeTicks), MaxTimeTicks = g.Max(x => x.MaxTimeTicks),

// This is the actual data Navigation = g.Sum(x => x.Navigation),

Debugging = g.Sum(x => x.Debugging),

Debugger = g.Sum(x => x.Debugger),

NavigationCount = g.Sum(x => x.NavigationCount), DebuggingCount = g.Sum(x => x.DebuggingCount), DebuggerCount = g.Sum(x => x.DebuggerCount),

};

StoreAllFields(FieldStorage.Yes); } }}

6 month token: ravendbconf2016

Questions?

Get in

TOUCH

Let’s do something great together.

info@corvalius.com

@Corvalius

/Corvalius

Av. Federico Lacroze 2352

7th Floor, Buenos Aires, Argentina

T. (+54 11) 4772-0650

c o r v a l i u s . c o m