Blog

Weather Data Transformation and Ingestion with Apache NiFi

Weather Data Transformation and Ingestion with Apache NiFi

This blog post describes how we transformed and ingested weather data using Apache Nifi.

17 Nov, 2017
Challenging the myth of the Data Scientist Unicorn

Challenging the myth of the Data Scientist Unicorn

This blog discusses how D2D CRC has seeked to understand the true meaning of a data scientist.

3 Nov, 2017
“What’s in the case” – Visualisation for Investigators

“What’s in the case” – Visualisation for Investigators

This blog post delves into the world of narrative visualisation and how it can be applied to investigations.

29 Sep, 2017
Now Hiring Andrews

Now Hiring Andrews

This blog explores the probability of duplicate names in an office of software engineers.

25 Aug, 2017
Big Data Architecture Principles

Big Data Architecture Principles

Architecture principles can provide a basis for decision making when developing solutions. This blog post describes Big Data-specific architecture principles used by D2D CRC.

18 Aug, 2017
Designing Big Data Applications

Designing Big Data Applications

This blog post provides a snapshot of the thought process we use when approaching a Big Data problem.

4 Aug, 2017
Related Term Suggestions with ConceptNet

Related Term Suggestions with ConceptNet

In this post we discuss some of the challenges we faced with related term suggestions and our approach to rapidly standing up a baseline solution before expending effort on more advanced solutions.

21 Jul, 2017
Agile Methodologies for Big Data Projects

Agile Methodologies for Big Data Projects

This blog post describes some of the challenges and the lessons learnt from using Agile on Big Data projects at D2D CRC.

5 Jul, 2017
Perceptual Hashing of Images

Perceptual Hashing of Images

In this post we will give a brief introduction to the concept of perceptual hashing of images, and demonstrate the simple technique we used.

16 Jun, 2017
Gaining Your First Data Science Job

Gaining Your First Data Science Job

Finding your first job out of University can be a daunting experience. It’s often difficult to know where to start or what skills and expertise employers are looking for.

16 Mar, 2017
Automatic Resource Management in Scala

Automatic Resource Management in Scala

The Beat the News team at Data to Decisions CRC (D2D CRC) have our heads buried (for the most part) in Spark for big data analytics. Naturally, we dabble a lot in the Scala programming language to support our Spark processing.

31 Jan, 2017