Storing user generated content

So it has been far too long since I have written anything useful on this page, mainly because of a change of job and therefore a decrease in time, I thought I would come back with a completely different topic, just for fun.

As part of the development of a platform my new company is creating, we came across a familiar question for tech start-ups these days:
"What database are we going to use to store all of this user generated content?"


Before it was a choice of a few vendors based on your scale and budget, but no more. You have other things to think about. Relational/Non-Relational? Records or documents? All of it has it's pros and cons which made our decision making process very hard.

The first thing people said to me was "It doesn't matter, in the end data is data and databases are databases". That tended to be the last time I spoke to those individuals. You know the type, they go to the tech meetups in London because they have a passing interest in tech and the square root of bugger all actual experience in building and integrating these things. Anyway I digress...

The first thing you should know is: There is no one right answer, its situational, its variable and it will probably involve more than one technology in the end to get the right result.

The first step we took was we needed to appreciate that what our requirements were:

• How is the data needed to be accessed
• How often do you write new data?
• How many concurrent reads?
• How often does data need to be updated
• Might the data need to be updated, if so how often?
• Is it a pure repository
• How easily does your environment have to scale
• How rapidly are you planning on growing? 
• What technologies play well with it
• What programming languages do you use? What libraries are available to you?

This is not an exhaustive list, but its a starter for 10 when you are evaluating your options because guess what, depending on your answers, The choice of solution should become more apparent.

The main decision we faced was: Relational vs Document store

One is great for data integrity, the other is super fast, and almost more importantly efficient, at reads.

Some great resources for Relational Vs Document store can be found here:
http://www.linuxjournal.com/article/10770
http://nosql-database.org/

Of course, even when you have chosen your path... you have to decide on what flavour of solution you want to go for... Your mySQL from your PostgreSQL, your mongoDB from your cassandra... etc

These again have their pros and cons, their advocates and their detractors. But once you have gotten that far, the rest of the journey is near complete.

As for us, you might have guessed by now that we went with mongoDB and that is what I will be blogging about over the coming months as i play around with it some more. I haven't stopped my MSSQL or SSRS adventures, just haven't anything to write home about at the moment.

Any suggestions on topics welcomed...

Tom