IBM Systems Magazine, Mainframe - November/December 2015 - 20

and its intent to offer Spark as a
service on IBM Bluemix*.
One of the key aspects of
Spark that is attracting a growing
following of adopters and
contributors is its strength as a
unification of the programming
interfaces for analytics. Spark
isn't only about data access, but
it's also about the framework
that's offered in terms of analytic
programming contexts. In many
ways, the analogy can be made
between what Spark offers for
analytics and how WebSphere*
and Java transformed application
development. Consider the
democratization of analytics
interfaces, abstractions over
underlying data access and
implementation specifics, as well
as the variety of programming
models to suit business needs.
Spark is fundamentally structured to provide independence
of application environments (see
Figure 1, right), which includes:
ĀÃ 6SDUNÃ&RUHÃWKHÃEDVLVÃRIÃWKHÃ
project, which provides task
scheduling, dispatching,
I/O and the programming
abstraction Resilient
Distributed Datasets (RDDs)
that are read-only distributed
collections of records that
can be stored in memory
or on disk. RDDs then offer
various operations that can be
performed on them including
transformations such as map,
union, intersect and actions
such as collect, count or
aggregate with a function.
ĀÃ 6SDUNÃ64/ÃDÃIXQFWLRQÃRQÃWRSÃ
of Spark Core that provides a
further abstraction of RDDs
used for accessing structured
and semi-structured data,
ZLWKÃDFFHVVÃYLDÃ64/ÃLQWHUIDFHVÃ
and access to sources of data
via JDBC and ODBC interfaces.
The data abstractions for
RDDs used in this context are

referred to as DataFrames
(formerly known as
SchemaRDDs).
ĀÃ 6SDUNÃ6WUHDPLQJÃVXLWDEOHÃIRUÃ
ingesting data and leveraging
available operations for RDDs
on ingested batches of data
ĀÃ 0/LEÃDÃPDFKLQHÃOHDUQLQJÃ
framework on top of Spark
Core that implements several
commonly used machine
learning algorithms (e.g.,
correlations, random data
generation, classifications,
regressions, decision trees,
clustering, etc.)

Figure 1: Apache Spark Structure

ĀÃ

*UDSK;ÃDÃJUDSKÃSURFHVVLQJÃ
framework and API enabling
efficient graph computations
with interactions across both
graph representations and
RDD representations of the
same data without duplication

With all of the challenges facing
enterprises and the advantages of
Spark, it's understandable why this
community of users and contributors has grown so significantly.
Even commercial vendors are now
offering Spark-based solutions for
key analytics such as fraud detection and customer insight.

Spark on IBM z
We hear a great deal about the
volume, variety, veracity and
volatility of data coming from
sources external to the enterprise,
such as social media, blogs, etc.
However, insight that can be
gained by combining the analytic
results from external data with
high-value (and highly sensitive)

Figure 2: Integration of Apache Spark Analytics With
Transaction Systems

20 // NOVEMBER/DECEMBER 2015 ibmsystemsmag.com


http://www.ibmsystemsmag.com

IBM Systems Magazine, Mainframe - November/December 2015

Table of Contents for the Digital Edition of IBM Systems Magazine, Mainframe - November/December 2015

IBM Systems Magazine, Mainframe - November/December 2015
Table of Contents
IBM Perspective: Opening new options for real-time analytics
Editor's Desk: Predictive analytics enhances purchases
Infographic: Breaking down the mainframe skills gap
Partner PoV: IBM z Systems clients must manage existing and emerging assets together to get the most value
Focus on Storage: The IBM Spectrum Storage family of software-defined solutions simplifies storage management
Currents: Architect real-time analytics into operational systems for competitive advantage
Cover Story: Dynamic Spark: IBM brings the Apache project's analytics power to Linux and z Systems servers
Feature: Personal Shopper: Retailer optimizes analytics to drive a more customized client experience
Tech Showcase: In a roundtable, distributors discuss how IBM LinuxONE enables flexibility for users
Tech Corner: Runtime Diagnostics and Predictive Failure Analysis V2.2 improve z/OS availability
Solutions: Strobe 5.2
Stop Run: IBMer volunteers time to educate people about malaria and technology
Reference Point - Global Events, Education, Resources for Mainframe
2016 Mainframe Solutions Edition Product Index
IBM Systems Magazine, Mainframe - November/December 2015 - Intro
IBM Systems Magazine, Mainframe - November/December 2015 - IBM Systems Magazine, Mainframe - November/December 2015
IBM Systems Magazine, Mainframe - November/December 2015 - Cover2
IBM Systems Magazine, Mainframe - November/December 2015 - 1
IBM Systems Magazine, Mainframe - November/December 2015 - Table of Contents
IBM Systems Magazine, Mainframe - November/December 2015 - 3
IBM Systems Magazine, Mainframe - November/December 2015 - 4
IBM Systems Magazine, Mainframe - November/December 2015 - IBM Perspective: Opening new options for real-time analytics
IBM Systems Magazine, Mainframe - November/December 2015 - Editor's Desk: Predictive analytics enhances purchases
IBM Systems Magazine, Mainframe - November/December 2015 - 7
IBM Systems Magazine, Mainframe - November/December 2015 - Infographic: Breaking down the mainframe skills gap
IBM Systems Magazine, Mainframe - November/December 2015 - Partner PoV: IBM z Systems clients must manage existing and emerging assets together to get the most value
IBM Systems Magazine, Mainframe - November/December 2015 - 10
IBM Systems Magazine, Mainframe - November/December 2015 - Focus on Storage: The IBM Spectrum Storage family of software-defined solutions simplifies storage management
IBM Systems Magazine, Mainframe - November/December 2015 - 12
IBM Systems Magazine, Mainframe - November/December 2015 - 13
IBM Systems Magazine, Mainframe - November/December 2015 - 14
IBM Systems Magazine, Mainframe - November/December 2015 - Currents: Architect real-time analytics into operational systems for competitive advantage
IBM Systems Magazine, Mainframe - November/December 2015 - 16
IBM Systems Magazine, Mainframe - November/December 2015 - 17
IBM Systems Magazine, Mainframe - November/December 2015 - Cover Story: Dynamic Spark: IBM brings the Apache project's analytics power to Linux and z Systems servers
IBM Systems Magazine, Mainframe - November/December 2015 - 19
IBM Systems Magazine, Mainframe - November/December 2015 - 20
IBM Systems Magazine, Mainframe - November/December 2015 - 21
IBM Systems Magazine, Mainframe - November/December 2015 - 22
IBM Systems Magazine, Mainframe - November/December 2015 - 23
IBM Systems Magazine, Mainframe - November/December 2015 - Feature: Personal Shopper: Retailer optimizes analytics to drive a more customized client experience
IBM Systems Magazine, Mainframe - November/December 2015 - 25
IBM Systems Magazine, Mainframe - November/December 2015 - 26
IBM Systems Magazine, Mainframe - November/December 2015 - 27
IBM Systems Magazine, Mainframe - November/December 2015 - 28
IBM Systems Magazine, Mainframe - November/December 2015 - Tech Showcase: In a roundtable, distributors discuss how IBM LinuxONE enables flexibility for users
IBM Systems Magazine, Mainframe - November/December 2015 - 30
IBM Systems Magazine, Mainframe - November/December 2015 - 31
IBM Systems Magazine, Mainframe - November/December 2015 - 32
IBM Systems Magazine, Mainframe - November/December 2015 - 33
IBM Systems Magazine, Mainframe - November/December 2015 - Tech Corner: Runtime Diagnostics and Predictive Failure Analysis V2.2 improve z/OS availability
IBM Systems Magazine, Mainframe - November/December 2015 - 35
IBM Systems Magazine, Mainframe - November/December 2015 - 36
IBM Systems Magazine, Mainframe - November/December 2015 - 37
IBM Systems Magazine, Mainframe - November/December 2015 - Solutions: Strobe 5.2
IBM Systems Magazine, Mainframe - November/December 2015 - 39
IBM Systems Magazine, Mainframe - November/December 2015 - Stop Run: IBMer volunteers time to educate people about malaria and technology
IBM Systems Magazine, Mainframe - November/December 2015 - Cover3
IBM Systems Magazine, Mainframe - November/December 2015 - Cover4
IBM Systems Magazine, Mainframe - November/December 2015 - RF1
IBM Systems Magazine, Mainframe - November/December 2015 - SE1
IBM Systems Magazine, Mainframe - November/December 2015 - SE2
IBM Systems Magazine, Mainframe - November/December 2015 - SE3
IBM Systems Magazine, Mainframe - November/December 2015 - SE4
IBM Systems Magazine, Mainframe - November/December 2015 - SE5
http://www.ibmsystemsmagmainframedigital.com/mspcomm/ibmsystemsmag/ibmsystems_mainframe_20200910
http://www.ibmsystemsmagmainframedigital.com/mspcomm/ibmsystemsmag/ibmsystems_mainframe_20200708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20200506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20200304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20200102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/2020mfse
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20191112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20190910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20190708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20190506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20190304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/relevantz_20190102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/2019mfse
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20190102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20181112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20180910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20180708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20180506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20180304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20180102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/2018mfse
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20171112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20170910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20170910_v2
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20170708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20170506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20170304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_sesupp
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20170102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_linuxsupp
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20161112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/MainframeSecurity
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20160910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20160708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20160506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20160304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20160102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20151112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150910_se
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150910
http://www.ibmsystemsmagmainframedigital.com/MFSkills
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150506_supp
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20150102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20141112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20140910_v2
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20140910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20140708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_gt_201405
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/BigData
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20140506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20140304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20140102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20131112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20130910_v2
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20130910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20130708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20130506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20130304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20130102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20121112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/buyersguide2013
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20120910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20120708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20120506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20120304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20120102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/ibmsystems_mainframe_2012bg
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20111112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20110910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20110708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20110506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20110304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20110102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20101112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20100910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20100910_bg
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20100708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20100506
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20100304
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20100102
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20091112
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20090910
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20090708
http://www.ibmsystemsmagmainframedigital.com/nxtbooks/ibmsystemsmag/mainframe_20090506
https://www.nxtbook.com/nxtbooks/ibmsystemsmag/mainframe_20090304
https://www.nxtbookmedia.com