Elastic – Data ingestion, storage, and visualization

Hey there everyone! I hope some of you are here following the talk I gave at YOLOcon. This blog post should be a good review of the topics discussed and will add links to resources that will help you get started in experimenting with Elastic.

For those who did not attend the talk, here’s a little background. I recently had the chance to attend Elastic{ON}, a conference for an open-source software suite called Elastic. The conference was absolutely fantastic, between the talks about future enhancements, real life use-cases, and geeking out with all these people who love the software as much as I do. Since I rarely hear people in our major talking about this suite, I’m taking the opportunity to share it with you all through this post. From my experience with the suite, I would summarize their main goals into the following: Data Ingestion, Storage, and Visualization.

When I first started working with Elastic, those three goals were represented in three software components, and it was referred to as the ELK Stack. These components were Elasticsearch, Logstash, and Kibana, and they remain the core of the suite today with a few new additions. Links to the main pages of each are below.

Logstash – Your primary data import tool

Elasticsearch – Your one-stop data shop

Kibana – Visualization center

Beats – Logstash’s little helpers

X-Pack – Enterprise upgrade

Cloud – Cloud hosting

Elasticsearch is the data storage component, and really the heart, of Elastic. In short, Elasticsearch is a distributed database that deals in indices of JSON documents. JSON documents are stored in indices, which are split and replicated across shards, which are spread across multiple nodes in your cluster. It is designed to store both structured and unstructured data. Essentially, if you can format data into JSON statements, you can put it in this database. Ingestion options include ‘PUT’ statements like the one below, or using Logstash and Beats to add data.

PUT /customer/external/1?pretty
{
“name”: “John Doe”
}

Logstash is your primary data collection tool. It has a huge list of supported inputs (53 in total) and facilitates the “any data goes” attitude. After choosing a set of inputs, you build configurations in your Logstash server to parse the incoming data into JSON documents or whatever other output format (pick from 55 options) you like. Assuming you go the Elasticsearch route, those documents are then added into your database!

Now, Beats works in conjunction with, and as a replacement for, Logstash. These are small data shippers that live on your local machines and either ship data directly to Elasticsearch or send it to Logstash for additional processing. There are a few of these officially supported, and you can read more about them online. There is also documentation on how to build your own custom beat, and there is a good chance that another community member has created and shared a beat for your use-case!

Filebeat – log files

Metricbeat – metrics

Packetbeat – network data

Winlogbeat – Windows event logs

Heartbeat – uptime monitoring

X-Pack is the set of tools that takes you from an open-source project to an enterprise-level application. It adds authentication and other security-related functionality, enhanced cluster monitoring, alerting functionality, scheduled reporting, enhanced graphing, and soon to be added machine learning abilities. All of these features are valuable as your take your project to an enterprise level, but I wouldn’t worry about getting your hands on them until then. You can find many free projects that mimic their capabilities. The integration won’t be as seamless, but they’ll save your pocketbook until you or your company has the funding for the supported versions.

Finally, Elastic has recently added cloud support into their list of features. They have partnered with AWS and Google Cloud to provide managed clusters for a subscription fee. This does allow you access to the X-Pack features. So, if you want to try it out on a budget, this may be your way to go.

The benefit to sticking with the core Elastic products is that you know everything will work together, and separate component updates are released in sync. But if you want to work with any of the X-Pack features and don’t have the budget, or if there are some other features you feel you need which don’t exist in the current release, turn to the community. There are quite a few replacements out there. Apache Kafka can replace Logstash to add caching and high availability to your data ingestion process. Grafana is a Kibana replacement that has built-in alerting. Or Elastalert can replace Alerting/Watcher from the X-Pack for that alerting need without replacing a core module. These are just a few alternatives that I’ve worked with, but many more exist.

In the end, if you have any interest in playing with “big” data at home or if you’re thinking about changing things up at work, you should seriously consider checking out Elastic. Their active community and quality documentation make it easy to ramp up with small home projects, while still providing professional services for enterprise implementations. If you’re looking for some inspiration to start a project of your own, look no further than Elastic’s use-cases page, here.

By – Anna Wendt

 

Advertisements

IASA Presents YOLOcon17

Hello everyone,

I would like to officially announce the Information Assurance Student Association’s second annual YOLOcon! This will be hosted on Saturday April 8th, between 9:30am and 2:30pm in Roosevelt Hall. This is a student driven conference that emphasizes student achievement. Most of our talks will be given by students, with the remainder given by industry sponsors. This unique opportunity gives students a chance to hear what their peers have decided to pursue as well as connect them with interested industry contacts who are looking to hire either interns or new graduates from the program. Lunch and t-shirts will be provided for all attendees.

Sponsors this year will include: Plante Moran, Quicken Loans, Snapchat, Consumers Energy, Electronic Brain Solutions, and Domino’s with more coming!

Any EMU student can register at ( https://goo.gl/forms/zG6Z2wga6XTzBTOD3 ) for free. Registration for a T-Shirt closes on March 17th. If you miss the deadline you can still come, however we will not provide a t shirt.

We are also encouraging all students to submit a proposal to speak at YOLOcon! We are looking for 15-30 min talks on any IA related topic. If you or a student you know would like to submit, please fill out the form here ( https://goo.gl/forms/YwhpMCSpgEmQK56e2 ). Please note, we can not guarantee talk acceptance into YOLOcon as we may receive more talks than we have slots available. Talks will be chosen on uniqueness and thoroughness of the submission. The last day to submit will be March 17th.

greenyoloconlogo2017

Thank you,
Jessica Wilson

IASA Presents YOLOcon!

Hello Everyone,

I would like to officially announce the project many of our students have been working on this semester: YOLOcon! This will be a student driven conference with talks and demonstrations given by students for students! We will be hosting it on April 9th in Roosevelt Hall on Eastern Michigan University’s campus from 10am to 2pm. Lunch and t-shirts will be provided.

Signup here! Must have a emich.edu address

yolo
Please join us for a full day of learning some amazing skills! We currently have talks set up such as “Host and create your own blog”, “An Introduction to Docker”, “An Introduction to Ruby”, and “Bash 101”. More talks will be announced soon, with topics ranging from offensive security to defensive tactics.

In addition to student talks, we will have a fun “chill out” room where anyone is welcome to practice the skills they have learned in any of the talks or have fun in our pre-made purple team environment! Lastly, we will also have a room dedicated to employers who want to hire the outstanding students who will be participating in this conference. This is your chance to talk to the industry and find your next internship or full time career!

We are all very excited to bring this to EMU and look forward to input!. If you would like to get involved with sponsorship, please feel free to contact me at jwils117 at emich dot edu.

Thank you,
Jessica WilsonPresident of the Information Assurance Student Association

 

IASA Winter Science Fair 2016

  • Upcoming
    • 2/3 Intern Fair
    • 2/9 Duo and Google “Debugging the Gender Gap” -Ann Arbor
    • 2/10 Ford round tables
    • 2/18 Mark Stanislav talks at Duo about hacking baby monitors
  • Science Fair
    • Joe – Ethics
      • Be smart!
    • Shane – One Line Hell (How to make an administrator flip a table)
      • Write a random byte in the kernel’s memory
        • Random system crashes and reboots
      • Drop random packets with iptables
        • Why is this even in iptables?
      • Turn off all cpu cores but one to slow a system down
      • Add a space to an important file/folder
        • /bin to /bin_
      • Etc. So many possibilities
      • How to stop these things?
        • Don’t let them get root
        • Don’t let them sudo
        • look in cron
        • Iptables
        • Know how to edit config files (in multiple ways)
      • Jessica – Reddit Reader
        • Used Python
        • Can grab reddit content for the user –even read comments
      • Stephen – Human Trafficking Investifation
        • Using public information to investigate human trafficking in Michigan and Ann Arbor
        • Has brought this to the attention of the Ann Arbor Police Department and they are now investigating the matter
      • Jeff Kaminski – Docker
        • A possible replacement to virtual machines — just that it isn’t quite ready yet
        • It can run applications rather than operating systems
        • Could be useful to test an application
      • Kent – WordPress Malware Deobfuscation
        • WordPress is the most popular website creator tool
        • Attack surfaces
          • WordPress and plugins
          • Attack surfaces
            • WordPress and plugins
      • Jeff – Block website tracking — Networkwide
        • Used a Raspberry Pi to help block websites from tracking usage
        • NoTrack DNS
      • Nikita – How to Master Social Engineering
        • Be ethical
        • Know your target
        • Act like a “salesman”
        • Can gather information on social media, background check, and corporate and personal websites
        • Make a good first impression
        • It’s easy for someone to know if your behavior is out of the norm — like if you’re nervous when you otherwise have no reason to be
      • Jacob – Virtualizing your attack environment
        • Helpful to practice for the upcoming competitions
        • VMware, Esxi, and vSphere
        • pfSense as a router.firewall
      • Alec – Fail2Ban Jails
        • Great way to keep suspected malicious visitors to the website from multiple attempts
        • Rules meant to detect known bad attacks
      • Yusef – IASA Computers for Children
        • Proposal for giving computers to children in low income families
        • One Laptop Per Child
        • Have hardware donated
        • Likely wont begin for multiple semesters

Kent’s Talk

  • Kent’s Ruby Talk
    • Using Ruby Scripts and MongoDB to analyze websites
      • His script/bot is able to analyze such sites as restaurant menu websites to get the contents of the site
  • Competition Practice
    • Go Team!

First Meeting of 2016

  • A big thank you to our guest, Denis Foo Kune, Ph.D — Leaking and injecting signals
    • It is possible to blast radio waves — energy — at a microphone to have it play specific audio
      • the wiring in electronics are sensitive to this energy
    • Might it be possible to apply this principle to a medical device such as a pacemaker?
      • Yes indeed!
    • Can the opposite be achieved?
      • It is possible to monitor computer activity by monitoring fluctuations in power levels
        • Encryption Keys can even be stolen
    • By observing patters in power fluctuations, versions of OSes/kernels can be identified
      • Windows 7 has an identifiable pattern in power fluctuations
      • Even specific YouTube videos can be identified
    • What Security Potential does this have?
      • It can be used to identify whether or not a machine is infected with malware
      • Malware is getting smarter, enabling it to more effectively hid from antivirus
      • However, power draws are often the same in versions of malware, even if it has been slightly modified in order to change its hash signature and thus thwart AV
  • Upcoming…
    • Ruby Scripting with Kent next week (1/15/2016)
    • Practices for ISTS and CCDC competitions coming up

CRYPTO-PARTY

  • Cryptography Talk
    • Modern encryption puts power in the people’s hands
      • prior, encoded messages were limited to the use of government/spies
    • Good ways to start using encryption
      • Signal (Formerly TextSecure), Tor Browser, orbot+orfox, browser plugins: privacy badger + noscript + disconnect + adblock +https everywhere + convergence
      • More advanced options: GPG, Whonix, Tails
    • (30 min of group learning of the above tools)
  • Next Week: Practice for CCDC/ISTS begins
    • Evil 101 (Red team tactics) – Calvin
    • Linux 101 – Jessica