Especially in larger organizations, computing often centers around dealing with large amounts of structured data. The Wolfram Language has a very powerful way to deal with structured data, using what it calls datasets.

A simple example of a dataset is formed from an association of associations.

data=Dataset[\[LeftAssociation]"a"->\[LeftAssociation]"x"->1,"y"->2,"z"->3\[RightAssociation],"b"->\[LeftAssociation]"x"->5,"y"->10,"z"->7\[RightAssociation]\[RightAssociation]]

The Wolfram Language displays most datasets in tabular form. You can extract parts from datasets just like you would from associations.

data["b","z"]

data["b"]["z"]

You can also just get the whole “b row” of the dataset. The result is a new dataset, which for ease of reading happens to be displayed in this case as a column.

data["b"]

Here is the dataset that corresponds to the “z column” for all “rows”.

data[All,"z"]

Extracting parts of datasets is just the beginning. Anywhere you can ask for a part you can also give a function that will be applied to all parts at that level.

data[All,Total]

If we use f instead of Total, we can see what’s going on: the function is being applied to each of the “row” associations.

data[All,f]

data[All,#x+#z&]

data[All,PieChart]

You can give a function to apply to all rows too.

data[f,"z"]

data[f,Total]

data[Max,Total]

You can always “chain” queries, for example first finding the totals for all rows, then picking out the result for the “b row”.

data[All,Total]["b"]

data["b",Total]

Particularly when one’s dealing with large datasets, it’s common to want to select parts based on a criterion. The operator form of Select provides a very convenient way to do this.

Select[{1,3,6,8,2,5,9,7},#>5&]

Select[#>5&][{1,3,6,8,2,5,9,7}]

The operator form of Select is a function which can be applied to actually perform the Select operation.

data[Select[#z>5&]]

data[All,Select[#>5&]]

Normal[%]

Many Wolfram Language functions have operator forms.

SortBy[{1,3,6,8,2,5,9,7},If[EvenQ[#],#,10+#]&]

SortBy[If[EvenQ[#],#,10+#]&][{1,3,6,8,2,5,9,7}]

data[SortBy[#x-#y&]]

data[SortBy[#x-#y&],Total]

Sometimes you want to apply a function to each element in the dataset.

data[All,All,f]

data[SortBy[#x-#y&],Total,#^2&]

Datasets can contain arbitrary mixtures of lists and associations. Here’s a dataset that can be thought of as a list of records with named fields.

Dataset[{\[LeftAssociation]"x"->2,"y"->4,"z"->6\[RightAssociation],\[LeftAssociation]"x"->11,"y"->7,"z"->1\[RightAssociation]}]

Dataset[{\[LeftAssociation]"x"->2,"y"->4,"z"->6\[RightAssociation],\[LeftAssociation]"x"->11,"y"->7\[RightAssociation]}]

Now that we’ve seen some simple examples, it’s time to look at something slightly more realistic. Let’s import a dataset giving properties of planets and moons. The dataset has a hierarchical structure, with each planet having a mass and radius of its own, and then also having a collection of moons, each of which have their own properties. This general structure is extremely common in practice (think students and grades, customers and orders, etc.).

planets=CloudGet["http://wolfr.am/7FxLgPm5"]

planets[All,"Radius"]

BarChart[planets[All,"Radius"],ChartLabels->Automatic]

If we ask about the moons of Mars, we get a dataset, which we can then query further.

planets["Mars","Moons"]

planets["Mars","Moons",All,"Radius"]

We can do computations about the moons of all planets. First, let’s just find out how many moons are listed for each planet.

planets[All,"Moons",Length]

planets[All,"Moons",Total,"Mass"]

planets[Select[Length[#Moons]>10&],"Moons",Total,"Mass"]

PieChart[%,ChartLegends->Automatic]

Get a dataset with moons that are more than 1% of the mass of the Earth.

planets[All,"Moons",Select[#Mass>LinguisticAssistant&]]

planets[All,"Moons",Select[#Mass>LinguisticAssistant&]][All,Keys]

Normal[%]

Catenate[%]

planets[All,"Moons",Select[#Mass>LinguisticAssistant&]][Catenate,Keys]//Normal

Here’s one more example, where we find the logarithm of the mass of each moon, then make a number line plot of these values for each planet.

planets[All,"Moons",NumberLinePlot[Values[#]]&,Log[#Mass/LinguisticAssistant]&]

Here’s how to make a word cloud of names of moons, sized according to the masses of the moons. To do this, we need a single association that associates the name of each moon with its mass.

WordCloud[\[LeftAssociation]"A"->5,"B"->4,"C"->3,"D"->2,"E"->1\[RightAssociation]]

Association[\[LeftAssociation]"a"->1,"b"->2\[RightAssociation],\[LeftAssociation]"c"->3\[RightAssociation]]

planets[WordCloud[Association[Values[#]]]&,"Moons",All,"Mass"]

For what it does, the code here is surprisingly simple. But we can make it slightly more streamlined by using @* or /*.

We’ve seen before that we can write something like f[g[x]] as f@g@x or x//g//f. We can also write it f[g[#]]&[x]. But what about f[g[#]]&? Is there a short way to write this? The answer is that there is, in terms of the function composition operators @* and /*.

(f@*g@*h)[x]

(h/*g/*f)[x]

planets[WordCloud@*Association@*Values,"Moons",All,"Mass"]

planets[Values/*Association/*WordCloud,"Moons",All,"Mass"]

As a final example, let’s look at another dataset—this time coming straight from the Wolfram Data Repository. Here’s a webpage (about big meteors) from the repository:

To get the main dataset that’s mentioned here, just use ResourceData.

fireballs=ResourceData["Fireballs and Bolides"]

GeoListPlot[fireballs[All,"Coordinates"]]

Histogram[fireballs[All,"Altitude"]]

Make a word cloud of the planets, with weights determined by their number of moons. »

Make a bar chart of the number of moons for each planet. »

Make a dataset of the masses of the planets, sorted by their number of moons. »

Make a dataset of planets and the mass of each one’s most massive moon. »

Make a dataset of masses of planets, where the planets are sorted by the largest mass of their moons. »

Make a dataset of the median mass of all moons for each planet. »

For each planet, make a list of moons larger in mass than 0.0001 Earth masses. »

Make a word cloud of countries in Central America, with the names of countries proportional to the lengths of the Wikipedia article about them. »

Find the maximum observed altitude in the Fireballs & Bolides dataset. »

Find a dataset of the 5 largest observed altitudes in the Fireballs & Bolides dataset. »

Make a histogram of the differences in successive peak brightness times in the Fireballs & Bolides dataset. »

Plot the nearest cities for the first 10 entries in the Fireballs & Bolides dataset, labeling each city. »

Plot the nearest cities for the 10 entries with largest altitudes in the Fireballs & Bolides dataset, labeling each city. »

What kinds of data can datasets contain?

Any kinds. Not just numbers and text but also images, graphs and lots more. There’s no need for all elements of a particular row or column to be the same type.

Can I turn spreadsheets into datasets?

Yes. SemanticImport is often a good way to do it.

What are databases and how do they relate to Dataset?

Databases are a traditional way to store structured data in a computer system. Databases are often set up to allow both reading and writing of data. Dataset is a way to represent data that might be stored in a database so that it’s easy to manipulate with the Wolfram Language.

How does data in Dataset compare to data in an SQL (relational) database?

SQL databases are strictly based on tables of data arranged in rows and columns of particular types, with additional data linked in through “foreign keys”. Dataset can have any mixture of types of data, with any number of levels of nesting, and any hierarchical structure, somewhat more analogous to a NoSQL database, but with additional operations made possible by the symbolic nature of the language.

Can I use datasets to set up entities and values for them?

Yes. If you’ve got a dataset that’s an association of associations, with the outer keys being entities and the inner ones being properties, then you just have to put this inside EntityStore, and you’ll pretty much have everything set up.

Tech Notes