What’s the Big Deal with Big Data? Your questions answered

‘Big Data’ - It’s a term that has gained a lot of momentum and as far as we can see, it won’t be slowing down anytime soon. It’s become such a big part of the everyday vernacular across industries that weren't always synonymous with tech. We know everybody is talking about it, so we sat down with our Head of Analytics - Gala Camacho-Ferrari - to answer the big three questions - What, How and Why, to get you up to speed and running on the nuts-and-bolts of this global hot topic.

Q. So, what is Big Data?

A. The definition is forever changing. What it literally means is: data that is too big to handle on the specific device you are on at any given moment. For example, your smartphone can only handle a certain amount of data and as soon as it becomes too much for that device, it becomes what we call Big Data. If you’re using a bigger computer then it might not be considered Big Data because the processing power is larger.

smart city.png
What it [Big Data] literally means is: ‘data that is too big to handle on the specific device you are on at any given moment.’

When the term is used appropriately in the present day, it tends to refer to data that is live. When you are acquiring new information every second, you can imagine how that quickly becomes overwhelming, no matter how strong your computer is.

Q. How has Big Data evolved?

A. Two things have happened recently:

  1. Devices have become more capable of managing Big Data, meaning the hardware is improving exponentially.

  2. The software advances mean that the algorithms are more complex, enabling Big Data to be analysed and accessible, making it easier and faster than ever before.

So, you can understand that ten years ago, Big Data would have been a different amount of data than it is in 2019. When you combine these two immeasurable improvements, you can see how it has aided this concept of actually using Big Data to inform all kinds of decisions.

Q. Why is Big Data such a hot topic?

A. The term Big Data has been around for a long time. More and more funding has been pumped into research and the creation of devices sophisticated enough to process large amounts of information. As these devices have become more accessible to people and businesses, so too has the concept of using Big Data.

We can now build devices that gather data live and quickly, and we have the ability to transfer that information live and quickly, therefore actually absorbing that data - saving it, making analyses on it - live. For a long time, the machines that could capture Big Data were so niche. Big Data was used in a smaller number of sectors, and now it's become mainstream, therefore, developing the benefits across more industries in greater capacities.

urban city.png

Q. Can you provide an example of the evolution of how we process Big Data?

A. Weather stations would create data to be collected once a week. Then it became once a day, twice a day, and can now be generated every minute - sometimes even, every second. Where someone once had to visit that weather station each time new information was collected, to download and create space in the device’s memory, we now have infrastructure that can transfer the data somewhere and software that can gather it and analyse it.  

Now the questions are: Are you saving that data? Are you analysing it as it becomes available? Both? The creation of algorithms and software to manage and analyse Big Data are more important than ever because we now have devices that have real-time data coming in, in all sorts of industries - space, weather, traffic, finance, the list goes on. These are all examples of Big Data and the market can only keep growing as this information becomes monetised by companies who have access to it, and companies who can analyse it.

Q. How can Big Data help us understand Cities?

A. When people talk about Big Data in cities, they are usually talking about movement data, but many times, people say Big Data when they are actually just talking about data. Cities need to make fast and responsive decisions about the needs of citizens. Often traditional data doesn’t cut it because it is outdated, or is not granular enough to give us the kind of insights we need.

For example, if you’re trying to stimulate the local economy in a main street by testing out different strategies and activations, 5-year-old data won’t help give you information about the impact you are having in real time and what is working or what is not. Social Data is derived from many different data sources. Many of these sources of Big Data collect information at an unimaginable rate: Facebook generates 4 new petabyes of data per day and back in 2008 we found Google was “processing a total of more than 20 petabytes of data per day”, users upload an average of 95 million photos to Instagram per day. Neighbourlytics pulls information and analyses Social Data to provide city-makers with real-time insights on the places they own and manage.

Transcript from an interview between Stephanie Marks (Writer) and Gala Camacho Ferrari (Head of Analytics) at Neighbourlytics Head Office, Melbourne, Australia, Wednesday 5th June 2019.