Big data is a name for data sets that are too large, too fast or too loosely structured for the tools built around a relational database. This page is an overview rather than a tutorial. I'll first look at what the term means and at the units used to measure data volume. Then we'll look at who generates big data and how people analyze it. The last sections note what has changed in the field since the page was first written and collect introductory videos.

• What is the big data ?
• Units of Data Volumn
• Who is generating big data ?
• How to Analyze the Big Data ?
• What has changed since this page was written ?
• Video Digest

What is the big data ?

Even though a lot of people are talking about Big Data, but it is not easy to clearly define what it is. Big Data is not a technical term, it is more like a marketing word. Like most of the marketing work, the term Big Data also sound like easy, but very difficult to clearly define it.

It may sound like 'Huge amount of data' and seems to mean 'Only size matters'. Size is an important criteria for Big Data, but it is not the only criteria for the big data. In addition to size of the data, the format of data, relationships among data, structure in the data set can also be important criteria to differentiate the conventional data and big data.

One of the definition that I have heard and I like is "Big data is a set of data that cannot be processed / analyzed by the conventional relational data base technology'. This sounds much technical and clearer to me.

In conventional data base, the first thing we do is to define a clear data structure (table structure) and all the data are placed according to the predefined format. So each and every data are placed under predefined label (e.g, column name) and most of analysis and process is done based on those predefine label. In other words, we already knows about the overall data structure and relationship between each of items in the database. But in most case of big data, huge amount of data are just collected and accumulated. They don't have any clear label, there are no predefined structure categorizing those data. (Of course, there may be very primitive/basic categories like 'text', 'image', 'video' etc, but these kind of labeling/category would not give you any specific information). So, in Big data splitting the accumulated data into a set of meaningful group becomes a very important steps for big data analysis.

A second description is common in industry. It lists three properties, often called the 3V: volume, velocity and variety. Volume is the size of the data. Velocity is the rate at which new data arrives, for example millions of events per second from a web service or a network. Variety is the mix of formats, such as tables, free text, images, video and sensor readings in one collection. Later versions add veracity, which is the uncertain quality of the data, and value. Both descriptions agree with the definition above. A data set becomes big data when one of these properties breaks the fixed table structure of a conventional database.

• Size alone does not define big data : speed of arrival and mix of formats matter as much as volume.
• The missing structure is the practical test : conventional databases need a schema before the data arrives, while big data is collected first and structured later.
• Grouping the data is the first analysis step : before any statistic makes sense, the data has to be split into meaningful groups.

Units of Data Volumn

Even thought size of the data is not the only criteria for Big data, usually very very huge units are used to describe the big data. If you read or watch things about Big data, you would hear many of data units that we don't normally use. For your convinience, I summarized some of the data units you would hear about in various big data discussion.

The two columns of the table are not the same size, and the gap grows with each step. A decimal gigabyte is 10⁹ bytes, while 2³⁰ bytes is about 7.4 percent more. At the yotta level, 2⁸⁰ is about 20.9 percent more than 10²⁴. This gap is why a disk sold as 1 TB shows up as about 931 GB in an operating system that counts in binary units. To avoid the confusion, the IEC defines separate names for the binary column: kibibyte, mebibyte, gibibyte, tebibyte, pebibyte, exbibyte, zebibyte and yobibyte, written KiB, MiB, GiB, TiB, PiB, EiB, ZiB and YiB. The byte names in the table then keep their decimal meaning. In 2022 the SI also added two larger prefixes, ronna for 10²⁷ and quetta for 10³⁰.

• Check which column a number uses : storage vendors and network rates use the decimal column, while memory sizes usually use the binary one.
• The IEC names remove the ambiguity : 1 GiB is always 2³⁰ bytes, and 1 GB is 10⁹ bytes.
• Each step is a factor of about one thousand : 10³ in the decimal column and 2¹⁰ = 1024 in the binary column.

Who is generating big data ?

In most of conventional data base system, usually we have a certain group of people who is generating the data and most of those people knows that they are now generating the data. For example, a lot of sales people is generating data by inputting sales records by keyboard, scan, bar code reading etc.

But in case of big data, everybody (every one of us) and even machine (like sensors and camera) are genering the data and in most case we don't even realize that we are now generating those data. Followings are only a few examples of big data generation

• All the text and images in Google
• All the messages you creates in Twitter
• All the text and images you post in facebook
• All the search words you type in search engines (e.g, google, bing, yahoo etc)
• Location information when you are getting access to internet, mobile communication system etc

Machines now generate a large share of this data without any person in the loop. A mobile network is one example. Every base station and core network function writes logs, counters and call traces. The UE also reports radio measurements, and MDT collects those measurements together with location. Operators analyze this data to plan coverage, find faults and predict load, and 5G adds NWDAF, a core network function whose job is to collect network data and provide analytics from it. See 5G AI/ML for how 3GPP uses this data.

• Most big data is a by-product : people and machines generate it while doing something else, so nobody designs its structure in advance.
• Machines often generate more data than people : sensors, logs and network measurements run all the time.

How to Analyze the Big Data ?

Big data analysis does not replace the familiar statistical tools. It starts from them and adds new tools on top of them. How do you analyze the conventional data ? Probablity the most common things you would do would be as follows :

• Plot the data along time, region, age etc
• Taking average, moving everage etc
• Doing regression based on various functions

These techniques would be used as tools in Big Data analysis as well, but there are many cases where it is not enough to apply only these techniques in Big Data. For example, let's assume that you have all the messages posted on Twitter, what would you do ? Of course, you can apply the conventional methods to this data as well. For example, you can plot the number of twitters for hourly and daily basis. But what if you ask 'what is the most popular topics in twitter ?' 'How a specific topic spread across people ?' To give this kind of answer from twitters would not be easy with conventional data base technology (mostly based on SQL). You would need special tools to enable you with super high processing power, flexible search algorithm and relation finding algorithm etc.

The most important technique for Big Data analysis is "Pattern Finding", "Pattern Recognition"

How do you do those pattern finding ? Is there any clearly defined algorithm applicable for all the big data ?

The answer is 'NO'.

The most important technique for Pattern finding/recognition is 'your brain'. It is better if your brain has a lot of knowledges in specific field. If it is the case, your brain would be the best tool for the big data in that specific field.

If you are an expert and had a deep experience in a certain area and just pouring a lot of information into your brain for a long time, your brain would create its own intuition and may spit out some meaningful conclusion/information on its own. In many cases, you don't know exactly how it came out but in many case it will be very useful anyway. Just trust your brain and make it sure that you put big enough amount of data into your brain for enough long period of time.

I know.. this does not sound technical. If you are more interested in technical issues about this, search topics about machine learning, graph theory being discussed in connection with Big data analysis. Take some time to watch the videos linked below and it can be a good starting point.

Actually the huge amount of Videos posted on YouTube is one of the typical example of big data, currently YouTube will categorize and recommend you those videos based on the text title labeled by the person who posted. As far as I know as of now, it is only human brain to understand the contents of these video data and extract the information as a whole. It would be excellent if a software (or any kind of machine) can extract the information from this kind of video/audio/image data as our brain does and this is the biggest topics being done in big data research (especially machine learning or Artificial Intelligence). I think there always has been great deal of improvement in extracting information from image or short audio sentence.

• Conventional statistics are still the base : plotting, averaging and regression remain the first steps on any data set.
• Pattern finding is the core task : the goal is to find groups, trends and relations that nobody labeled in advance.
• Domain knowledge guides the search : an expert knows which patterns matter and which are noise.

What has changed since this page was written ?

The sections above were written when Hadoop was the standard tool and when computers could not yet understand the content of images and videos. Both statements have changed. So let's note the main changes, and read the older sections and the video list below with them in mind.

The largest change is in the analysis of images, video, audio and text. Deep neural networks now classify images, recognize speech and describe the content of a video frame. So the task that the previous section leaves to the human brain is now partly done by software. Video platforms now use the content of the video itself, and the viewing behavior of users, in addition to the title text. See the Machine Learning pages for the methods behind this.

The tools have also changed. Hadoop MapReduce writes intermediate results to disk between steps. Apache Spark keeps them in memory, so many workloads moved to Spark. Much of the storage and processing has moved from in-house clusters to cloud services. Finally, regulation such as the EU GDPR now limits how personal data may be collected and analyzed, which affects the social media and location examples above.

• Machines now read unstructured data : deep learning turned images, audio and text into data that software can analyze.
• The pattern finding is still the goal : the tools changed, but the task described above did not.
• Personal data carries legal limits : collecting it is no longer only a technical question.

Video Digest

The videos below are grouped by type, from short introductions to lectures and tool tutorials. They were collected when this page was first written. So some links may no longer work, and some of the tools they describe have been replaced as the previous section explains.