keeping it tidy - how to warehouse your sharepoint data

John WhiteChief Technology OfficerUnlimitedViz Inchttp://whitepages.unlimitedviz.com@diverdown1964

Keeping It TidyHow to Warehouse Your SharePoint Data

John WhiteCTO/Co-Founder of UnlimitedVizSharePoint Server MVP, SQL Server v-TS

jpw@unlimitedviz.comhttp://whitepages.unlimitedviz.com@diverdown1964

3© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Agenda

SharePoint Data Storage Data Warehousing Moving SharePoint Data Surfacing Warehoused Data in SharePoint Q&A

SharePoint Data Storage

The Problem

5© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Excel Data Storage

List based data Powerful Features Massive adoption Good usability

NOT a database Not relational Capacity limits Not authoritative

Excellent BI Client Surface the data.. DON’T store it

SharePoint Content Databases

7© DEVintersection. All rights reserved.

http://www.DEVintersection.com

SharePoint Content Databases

~160 fields

8© DEVintersection. All rights reserved.

http://www.DEVintersection.com

SharePoint Capacity Boundaries

Boundaries and Limits - http://technet.microsoft.com/en-us/library/cc262787 200 GB per content DB (general usage scenarios - or up to 4 TB) List view threshold – 5,000 simultaneous items List view lookup threshold – 8 join operations per query Items per content database – 60 million

Demo – WorkflowHistory List

Topic

Data Warehousing

ONE DOES NOT SIMPLY

RETURN HIS RAW DATA FROM THE DATABASE

12© DEVintersection. All rights reserved.

http://www.DEVintersection.com

BI Architecture 101

Data Marts

Extract, Transform, and Load (ETL)

Middleware Server(s)

DataWarehouse

Storage Design and Visualization

Data Cubes and Tabular

Models

E

T

L

Reporting Server(s)

BI and Designer Clients

Source data

13© DEVintersection. All rights reserved.

http://www.DEVintersection.com

EE

Microsoft enterprise (classic) BI

SQL Server DBSQL Server Integration Services (SSIS)

SharePoint (with)• Excel Services• PowerPivot for

SharePoint• SSRS SharePoint Mode• PerformancePoint

SQL Server DB

Storage Design and Visualization

SQL Server Analysis Services

Multidimensional and Tabular modes

L

SQL Server Reporting Services (SSRS)

ExcelSQL Data ToolsReport Builder3rd party tools

ETL

E

T

Source data

14© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Team BI and SharePoint Dashboards

Power Pivot Worksheets• Pivot Tables and

Charts• Power View

Data Marts and other

Data Cubes and Tabular

Models

Standard Worksheets• Pivot Tables and

Charts

PerformancePoint Reports• Analytic Charts and Grids• Decomposition trees

SQL Server Reporting Services Reports• Standard• Power View

PerformancePoint Scorecards and KPIs

15© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Why Warehouse?

Optimize for End Goal Multidimensional Flattened

Usability Reduced Load on Source Systems Security Requirements Licensing Requirements SharePoint (Performance)

Topic

Moving SharePoint Data

17© DEVintersection. All rights reserved.

http://www.DEVintersection.com

ETL Tools

SQL Server Integration Services Comes with ALL Editions of SQL Server (except Express)

SQL Server Data Tools http://www.microsoft.com/en-ca/download/details.aspx?id=36843 (For VS2012)

SharePoint List Source and Destination (Codeplex Project) http://sqlsrvintegrationsrv.codeplex.com/releases/view/17652 (Download) http://

dataqueen.unlimitedviz.com/2011/06/how-to-use-a-sharepoint-list-as-a-data-source-in-your-ssis-package/ (How to)

OData Source for Microsoft SQL Server 2012 (Official) http://www.microsoft.com/en-us/download/details.aspx?id=42280 (Download) http://whitepages.unlimitedviz.com/2014/03/using-the-odata-source-connector-with-sharepoint-o

nline-authentication/ (How to)

http://www.microsoft.com/en-pk/download/details.aspx?id=35585 – Required for Office 365 authentication

Moving SharePoint Data

Demo

Topic

Accessing Warehoused Data

20© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Data Consumption

Business Connectivity Services External List All SharePoint Editions (yes..Foundation)

SQL Server Reporting Services Advanced Reporting All Editions of SQL Server (even Express!) ..except Express doesn’t do SP Integrated Mode

Excel/Power Pivot Excel Services requires Enterprise SharePoint PowerPivot for SharePoint requires SQL Server Standard or better

Accessing Warehoused Data

Demo

22© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Considerations

Transformations Mashups Scheduling Data latency

In Conclusion

24© DEVintersection. All rights reserved.

http://www.DEVintersection.com

Q&A

@diverdown1964whitepages.unlimitedviz.com