Data In, Elasticsearch Out

Abstract: In this talk, I’ll give a whirlwind tour of how to get various common sources of data into Elasticsearch using Logstash and Beats. Once we’ve covered getting data in, we’ll take a look at filtering and enrichment options both pre and post ingest with Logstash and Elasticsearch and then finally at management, maintenance and best practices of storing data in Elasticsearch.

Drupal Content Analysis: Elastic is here to help

Abstract: There have been sessions around Elastic stack in devop world. The ELK (Elasticsearch, Logstash and Kibana) has become a great tool for devops to track the server performance and site performance. And from the name, people would consider Elasticsearch as a searching engine and tend to compare it with Apache solr a lot. But we are here to talk about how we can leverage Elastic software in Drupal content analysis.

Drupal is always adopted by large organisations to manage their content, and in most of the cases, a lot content. Do you know what’s in your site? We get this question asked a lot. Surely by using Drupal’s taxonomy and views system, we can organise the content in a logical way, but content analysis is not what views is designed for. Examples will be given in this session on using Elastic Kibana to visualise your content and find out the ‘hidden rules’ in your content. We will also talk about the best possible ways to integrate Drupal content with Elastic stack.

Building an Elastic IOT Pipeline

Abstract: This talk will take a quick whirlwind tour of the open-source products that are part of the Elastic stack, with a focus on Beats, which are light-weight data shippers written in Go. These Beats are ideal as IoT data producers/consumers. We will look at how you can write your own Beat relatively easily with our “beat generator” and then we will look at a bunch of different deployment scenarios. At the end of this talk, you will hopefully be able to get started writing your own Beats or using existing ones and ultimately begin building your dream IoT data pipeline with the Elastic stack.

Processing and Visualising Hong Kong Government Open Data With Elasticsearch

Live Workshop Presentation

Abstract: In the IoT space, you can expect to be generating significant amounts of data at high rates. It’s therefore important to have a flexible processing and visualisation pipeline for that data so you can examine and manipulate it in real-time. In this workshop, we will explore creating a mock data processing pipeline using open data from the Hong Kong government. What this workshop hopes to show you is how easily you can get up and running with the Elastic stack and visualise and explore your data in real-time. Ultimately this workshop should provide you the foundations to expand the example pipeline for your own data or other open source data you find.

Pingbeat: Y’know, for pings!

Abstract: Ping, it is your go-to tool for diagnosing networking issues. But what is a ping actually doing and what is it telling you? What if you could keep a record of ping responses across your network to look at historical issues and potentially predict upcoming problems? In this talk I’ll give a quick overview of the venerable and beloved ICMP ping. I’ll then introduce Pingbeat, a small open-source program written in Go that can be used to record pings to hundreds or thousands of hosts on a network. There are many existing tools out there similar to Pingbeat, but its power lies in its ability to write the ping response to Elasticsearch, an open-source NoSQL-like data-store with powerful, built-in search and analytics. Combined with Kibana, a web-based front-end to Elasticsearch, you get an interactive interface to track, search and visualise your network health in near real-time.

The Power of Open Data With ELK

Co-hosted Workshop

Abstract: Our tutorial session will enable participants to run up an ELK stack to process a dataset from data.gov.au, and leave them with the skills to apply the stack to nearly any dataset. The ELK stack, which is comprised of the open-source software projects Elasticsearch, Logstash and Kibana is a powerful data and analytics platform that makes it easy to take structured and unstructured data and present it in beautiful ways. We’ll take a sample dataset from data.gov.au, feed it through Logstash, store it in Elasticsearch and then create a dashboard of visualisations from the data in Kibana.