Data Quality and its challenges

I have just read that a new postcode area, E20 will be added to London ahead of the Olympics in 2012.

The postcode will be applied to the Olympic villiage and the surrounding area.

Now postcodes change all  the time and its nothing new, but for a long time, these changes are not flown through into utilities companies/marketing systems databases (if they ever make it) and as such there will be a gap where people in these postcodes will face challenges in getting services from these companies.

Keeping data upto date is a challenge but important for the consumers experience from your SQL led databases.

If you run a consumer facing database that you market from help is at hand:

http://www.qas.co.uk/ - These guys have seen it all and done it all, one of the major players in the market.

If your not already under the management of a similiar company and you dont have a huge team of R&D guys peddling as fast as the industry changes... it could well  be worth the time talking to them.

Initial dabblings in SharePoint

As part of our ever expanding Business Intelligence drive at work, we are trying to surface management KPI's from our datamart, to the various different businesses that we own.

Some initial hurdles:

• Our SQL Server is part of a cluster at our HQ, with no direct links to the outside world
• Each business has its own Active Directory as they were aquired over time (sound familiar?)
• We have a Sharepoint 2007 Server at our HQ, with visibility of the SQL Server

This poses some interesting chalenges:

• How do we suface the data on a self service basis to these companies?
• How do we maintain a robust security model?
• How do we limit data duplication and issues with master dataset management?

So far we are using dummy AD acounts at the HQ with VPN access... but its far from pritty...

Any thoughts on a napkin, welcomed!!

This might be one for the cloud...

Datamarts for reporting

The team today released our "Version 1" of our reporting datamart.


What is a data mart? (I hear you cry)
The principle is that rather than having reports for management running of our production analysis platform, we create a data set that is bespokely created to answer our managements reporting needs.


In short, a datamart is an abstract view of your database, with aggregates and summarisations that are required for repetative tasks, already built into its design.


Why you would need a data mart 
Benefits:
1. Fast response times to often complex reports
2. Quicker time to implement new reports by expanding an already existing model
3. Complete control of the business rules from a central location
4. One system to optimise for reporting purposes
5. Decreased impact of reports on your production environment


Slight drawbacks 
1. If pre-aggregated data is needed at multiple aggregation levels, all these levels would have to exist exist in the data mart
2. The data mart is only as good as the thought put into it


Considerations for design 
1. What is the purpose of the data mart? - How "Little" data does it require (be brutal)
2. The reports that it will generate, what data will it need?
3. How timely are these reports needing to be - this will feed into how often it will have to be produced
4. Is a cube is SSAS more suited? (How often does your data model change)
5. What technologies will sit on top of this platform? SSRS, qlickview, etc?


I realise this is not very detailed, but its a starter for 10.


Details will come in the next few days of what it looks like, and how it has been put to use...


If you want to talk about it in more detail:
You can tweet me at @tristix or email me at tristix@gmail.com