All about Data

We have heard a lot about data technologies and tools. There is data modelling, data quality, metadata management, data governance, data stewardship, data lineage, data profiling and so on. Let's figure out just HOW does all these concepts fit together. So basically it means this -  We need to understand the data we have , design and arrange it in a logical manner, centralize the pool of data and then map those data to its business definition by enforcing a standardization and governance process. When we have those in place, we can further understand the quality of the data by setting up the business rules to check the data correctness and  subsequently trace it back to the original source. This process is a continuous journey and the further we go, the better our data quality and governance are, and yes corporate jargon call this as Data Strategy.So when we have all these concepts and understanding in place aka the Data Strategy, we can choose the solution from one or a combination of tools from vendors like SAP Power Designer, Information Steward, Informatica, Erwin , ER/Studio, Collibra that meet our budget and IT environment to implement it.

SAP Design Studio - Developing Extensions Using the SDK

Reality of Business Critical Reports

Recently I had a few sessions conducting asessment across the Support team to understand their concerns and criticality of the support procedures. I threw in questions like what if the BI system were to decommision now, what they forsee that will trigger a huge impact? And to sum up all the conversations that took place, there are many reports supported but the most critical ones being the financial management reports that use the BI platform for business view mapping and data transformation. The usage of these reports had become an integral part of the support mechanism as every month end of financial report submission, there is already a process in place that involve both IT Support and business users in area of data validation and tracing, mapping upload and changes all the way to rectifying posting issues when there is discrepencies reported in BI report total sum value when compared to the operational reports. Hence the criticality and success of BI is actually largely supported by the ongoing process that involve both IT Suport and business that enable the usage of it to be an integral part of bau.

Delta Load Implementation

As ETL tool such as Data Services and SSIS evolve, we can see there are request to extract SAP ECC tables directly out from the source (without going through SAP BW). There are a lot of considerations when we land on that approach mainly around the corporate datawarehouse roadmap and data policy, the standard EAI layer and the delta load mechanism.

When this approach is adopted, an organization should evaluate their existing datawarehouse such as SAP BW and the implication of single source of truth when data is extracted out into another system. Is the staging system supposedly to be a global datawarehouse in long run, is it planned to be the regional datawarehouse layer or just to cater for silo project as a 'dump' or staging area to transform some data to meet silo business unit reporting needs?

On the technical side, I always put emphasize on the delta mechanism as there is quite an in depth configuration at the ECC end. The common delta method may be timestamp and there are certain tables that store the timestamp records when transactional records were added. There are also log table to detect changes or deletion in records. And during data load there is additive or overwrite records functions. Another interesting area to take into account is the initial load. Some huge extraction from multiple tables with complex join require setup tables to populate huge historical dataset first before the delta take place. This is a pointer noted for performance consideration. 

I came across design using Data Services to extract data directly from huge ECC tables that prior to loading read from an ABAP program which identify the delta mechanism or logic such as reversal of posting etc. Any approach that the organization takes for data extraction, it is good to cover all the area during IT strategy or solution stage from BI roadmap to the technical and security assessment of the solution.

P/s this is a good read on delta methods

The Foundation Must Be Right

We hear a lot about the power of business intelligence through predictive analysis and data mining, the need of dashboard for overview and decision making and the empowerment of operational users through self service reporting. These are outcome of a successful BI practices supported by mandatory pillars - BI Framework , Data Foundation , Business Driven Values.

BI Improvement

I was assigned to lead the BI improvement task in my organization and I started off with a very detail study of the existing BW environment from the start of the system to date. To tell a technical story is always based on facts and statistic.To link statistic to improvement is a challenge. There are thousands of BW objects and to start is a challenge itself.

So I started out by accessing the current state from three perspectives -> Common Shared Layer, Data Integrity and Usage. I tracked down the objects in a chronological manner by years and projects. To determine common shared objects and layers, I go into the main multiprovider that zooms down to the datasources. I listed down all the datasources and the direct propagation layers. Next I listed down all the main infoobjects use in the main query and map it to the business context. And I checked on the last access date of the query. 

With this analysis results, I am able to derive the statistic on the number of projects that do not shared the common layers and objects to prove that the projects run in silo, how many master data with the same business context are created to prove that there is no single source of truth for reporting within the same ECC module and the last access of the queries to prove the BI usage. This analysis will act as the current state to measure how much we are able to improve by comparing the same statistics after implementing the improvements plan.

And how does this relate to cost saving over the time? That is my next challenge to come up with facts and findings that can tell this 'story' from financial perspective. 

Data Lineage in SAP Environment

It is great to hear that SAP metadata management has the coverage of end to end SAP environment that leverage on Information Steward features such as Metadata Management and Metapedia. But the tool itself cannot be a standalone feature when it comes to the  purpose of data lineage initiatives that is to ensure the data accuracy and data standardization. We still need to enforce a process to govern the custom master data metadata in BW environment that must be rooted to the source of master data in ECC.

This is a good read on Metadata Management in SAP Landscape.

A Look at SAP BI Servers Landscape

In many organizations that is going into the maturity stage of BI, there will be additional service offerings brought into the business and as the technology evolves. Many already embarked on Business Object 4.1 as their BI frontend reporting tool and SAP BW as their datawareouse. When the needs of Data Services and Information Steward arises for the purpose of data profiling and cleansing, the existing landscape has to be scalable to ensure that there is always the first emphasize made on single source of truth for reporting and this is very much refer to data structure and master data. Below is an example of the architecture landscape proposed on the aforementioned.

Please also note that the BO 4.1, DS and IS servers share the same server but with own repository and services. Architecturally, IS and DS are inseparable in that IS relies on DS.  In addition, they have a lot in common as well, they both leverage Information Platform Services (IPS).  Information Steward and Data Services both rely on CMS services for centralized user and group management, security, administrative housekeeping, RFC Server hosting and services for integrating with other SAP BusinessObjects software (i.e. BI Launch Pad).

For more information:
http://scn.sap.com/community/information-steward/blog?start=0

Relating Visual Analytics with daily life

Recently we have implemented Good Service Tax of 6% and this is how we can view it from a Heatmap, a way we can illustrate visual analytic in our daily event.The space indicate the proportion of the GST to monthly salary, the intensity of the color depicts the volume of the figures, the higher or bigger the darker the color.

Cloud Computing... I am coming

Lately I'd been bombarded with the 3 famous technical jargon - IaaS, PaaS,SaaS. Depending on the vendors, for SAP they put it as IaaS, DbaaS, PaaS.

As we mature into the cloud technology, it is really the time to think about a plan to adopt this robust framework and more importantly is to plan out a strategic transition phase to move from on premise platform to the cloud. There will also be an intermediate phase in which we need to factor in the extension of existing on-premise solutions to the cloud solutions.

It is exciting to imagine that in the future everything in the future is accessible from the cloud and there is nothing that we need to install or stored locally anywhere in our own devices. We can tap on to any device and start to work! We can use any applications through subscription base and in a way allowing deployment to be more agile and flexible.

Governance and Enforcement

Governance as we always hear it is nothing new at all in IT world. We create a set of standards and conventions in order to ensure things can be done more efficiently in long run and minimal risk of downtime incur to BAU. New developments or changes has to go through approval board, some call it CAB. However most approval process are merely a decision based on written regulation in the big governance book and not made to help the business achieve the goal of having governance at the first place - that is to enable things to run more efficiently in long run and ensuring minimal risk of BAU downtime.

This in turn result in the inefficiency of the governance framework in an organization as it is treated as enforcement rather than governance. A lot of governance framework in an organization is not revised over times and this contribute to the inefficiency of governance in an environment that is fluid and change over time. In BW project, a lot of project consultant finds it hard to perform after go live support that require immediate fix as they have to go through rounds of approval for transports and they are not allow to directly create objects in Production environment that support emergency data load such as infopackage. 

At the end, if the rules in the governance book is not bent to suit the business needs and its urgency, this is merely blind enforcement and not a practical way of governing a system. A lot may argue that if we allow it to happen one time, it will happen again. So in short , it is up to the governance board to exercise their power in tackling difficult situation and that is when in depth knowledge, experience and exposures are important criteria for an effective governance authority.

R - A Cool Guy with Serious Thoughts on Enterprise Technology

I got to know this cool guy 'R' who is an expert in telling you how disruptive technologies and new business models can impact the enterprise. I guess this is the kind of guy a lot of big organizations need to engage with before embarking on a serious of new technology adoptions. And not directly engage with Products vendors who is more bias in their own offerings.

Hi R,

I stumbled on your very good blog at http://blog.softwareinsider.org and here’s my POV.

Firstly, many organizations are still not mature in terms of BI and how it can really bring in the ROI. They started with BW as the backend and Bex query, Analyzer and WAD as the frontend. Initially dashboard is not that impressive when it was deployed from WAD so BO come into place with Xcelsius. Although Xcelsius can produce some impressive features, it is not really integrated with BW objects as compared to WAD and maintenance is very high. Most time was spent to develop the dashboards rather than looking into the data and business values. Over the next couple of months , we started to hear about Project Zen which eventually become Design Studio that looks something like WAD but with more features such as HTML5 and mobile integration (MOBI). We also have Advanced Analysis and BO Explorer that can directly consume data from HANA model.

As we continue with the technology advancement, we start to hear about Lumira as a Visual Storytelling tool. It is bundled in SAP HANA license so having HANA is mandatory. Lumira can produce very fast and interactive self-service dashboard when it consume directly from HANA model. But wait, remember in Gartner Magic Quadrant we have Tableau that rank the highest in data visualization and BI space. And even SAP has plug-ins ready for HANA to be consumed directly in Tableau. So there goes another debate for some companies, Lumira or Tableau.

Apart from that, not to mention there is also EPM tool for some power users who access the financial consolidation report. Imagine what the user will be getting when he opens his Excel early in the morning – he sees Bex Analyzer for reports which is not migrated yet , Advanced Analysis  for new reports and EPM for Consolidation and Planning reports. They also have the web based reports in BO portal (Webi and Xcelsius), self-service reports in BO Explorer and some old WAD and Bex reports in Enterprise Portal. In real life scenario, when we introduce new technologies, it does not happen in a clean cut manner but  through  gradual adoption and migration. All these incur cost and time. At the end, the management will ask the questions, is there any alternative besides getting in HANA, BO, Lumira etc. As for end users, they will be confused with so many tools they need to adopt.

On the application development site, how HANA become a game changer is through SAP River and Eclipse platform. So imagine data consume directly from HANA straight into web application layer on Eclipse using SAP River, what is the ‘new’ direction for application development now ? Do we shift from SAP platform to Eclipse platform now? Webdynpro ,workflow and ABAP to JAVA & SQL?

When HANA model become the major part of data acquisition for all the tools above, no doubts HANA Studio is going to be the new ‘RSA1’. And guess what, many BW developers out there are struggling to get a project that can give them the opportunity to pick up skills in HANA Studio.

In terms of BI, it is not only about technology advancement but also about the governance of data quality in the organization. So when the technology advances, most companies still struggle with the data quality or sap ECC convergence into global instance. To have a successful BI, these two main factors has to complement each other and be executed at the right track by the right party in the holistic BI roadmap.

Ok, back to real questions – HANA, so what does all these has to do with HANA and the direction? Simple. All of the new technology and tools are geared towards the HANA platform. It does not only change the company BI strategies but the People, Process , Innovation Trends and Jobs . So you can see now how big the impact is to the world out there with the new direction.

There you have it my POV. I am more on an enthusiast on the whole SAP BI innovations and how it evolves over the time, keen to gain and share and learn from feedbacks and POV from expert like you!