Elastic {ON} Tour Conference, November 2015

By Rik Palmer

Posted on

This autumn the team at Elastic are on tour. Elastic have grown enormously over the last few years, and they now offer a whole stack of solutions to assist in providing lightning fast data search, statistics & analysis, graphical data representation and much more besides.

As extensive users and advocates of Elastic and its related products (such as Logstash and Kibana), Sarah and Rik made their way to London Bridge to hear what’s new in the world of Elastic.

ElasticStageView

The first thing to explore were the demo stations, showcasing some of the latest graphical ways to represent the data stored in Elastic indices. These impressive visualisations provided an insight into the many ways for businesses to provide their valuable data to customers

Also dotted around the venue were “Ask Me Anything” stations, where attendees could discuss their burning questions with experts in the conference breakout areas. A great concept, although the Elastic team were somewhat overwhelmed by the number of attendees keen to pick the brains of their experts!

Morning Session

The morning kicked off with a talk from Elastic’s CTO Shay Banon, who revealed the inspiration to create ElasticSearch; to devise a tool for his wife to search for cooking recipes! Shay talked further about the history of Elastic and how it has grown to become a fundamental part of the technology stack for organisations including Microsoft, Wikipedia, Facebook, eBay, GitHub, The Guardian, Reuters, Dell, Goldman Sachs, Adobe and many more.

The morning of the conference focused on the Elastic roadmap and introducing the new features of ElasticSearch 2.0, Kibana 4.2, Beats, Watcher, and Found. Some of these were new to us, although we are very familiar with the ‘ELK’ stack at Makemedia (Elastic Search, LogStash and Kibana). Below are some of the useful tools and services owned by Elastic.

  • Kibana
    Kibana is an open source data visualisation plugin for ElasticSearch. It provides visualisation capabilities on top of the content indexed on an ElasticSearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.
  • LogStash
    A tool for collecting, processing and forwarding events and for logging messages and storing them in Elastic.
  • Marvel
    For monitoring and managing the status of ElasticSearch and troubleshooting problems by viewing how many documents are being indexed, and how many searches your cluster is serving.
  • Watcher
    A plugin that allows you to trigger notifications based on changes in your data.
  • Shield
    A commericial Elastic product that makes it easy to add enterprise-grade security to the ELK stack, allowing for role-based access control, LDAP-Based Authentication and Encryption, audit logging and IP filtering.
  • Found
    Providing hosted and manager ElasticSearch as a service.
  • Packetbeats
    Network packet analytics posted directly to Elastic.
ElasticCrowdView

Afternoon Session

The latter part of the day was focused on real life case studies, with organisations sharing how they’ve implemented Elastic for their own products.

The Guardian

Graham Tackley from The Guardian shared an insightful walkthrough of the tools they have built with the ElasticSearch stack to provide real-time media analytics, processing up to 40 million documents a day.

  • CMS for Editors: A CMS that allows editors to create content, indexed in Elastic for searching.
  • Ophan: An analytics system for viewing who, where, and how long users are browsing articles.
  • The Grid: An image library for editors to discover relevant photography to support their writing.

Goldman Sachs

The Goldman Sachs technology division has put ElasticSearch to use in several innovative ways. These include an application to assist their legal department with contract searches, enabling executives and clients to track trades, and to assist engineering teams in locating and eliminating software bugs.

Pipeline Aggregations

An important topic of the day was the introduction of pipeline aggregations in Elastic 2.0. As an extension of the existing ES aggregations framework, it provides a number of computation types that users can perform on top of the standard aggregations results.

Hadoop

Although not Hadoop users, it was great to discover more about it at the conference. Inspired by Google’s MapReduce, Hadoop makes it possible to run applications on systems with thousands of nodes, often with thousands of terabytes of data. It’s distributed file system facilitates rapid data transfer rates among nodes, allowing the system to continue operating uninterrupted in case of a node failure. This approach lowers the risk of catastrophic system failure, even if a significant number of nodes become inoperative.

ElasticSarahAndRikDuring the course of the day we met a variety of users within the Elastic community, discovering interesting use cases for the technology. Household names including Netflix, Facebook & Microsoft are using Elastic to build out their search capabilities, but it was interesting to learn about some of the more unusual use cases. NASA is using Elastic to store and analyse data sent back from the Mars Curiosity Rover, and the Victoria and Albert Museum plan to use it to analyse visitor data to determine how they could optimise the placement of their artefacts within the museum.

Elastic On Tour was a useful and informative conference, and we returned to Brighton full of ideas about how we would integrate some of Elastic’s latest developments into our solutions at Makemedia.

 

Back to top