Big Data Security Analytics Framework

This project is maintained by The OpenSOC Project


The OpenSOC project is a collaborative open source development project dedicated to providing an extensible and scalable advanced security analytics tool. It has strong foundations in the Apache Hadoop Framework and values collaboration for high-quality community-based open source development.

The OpenSOC project has the following goals:

  • To provide a collaborative open source community for development of an extensible and scalable advanced security analytics tool
  • To encourage open communication for additional features and identification of deficiencies for a stable and functionally usable tool
  • To identify key feature enhancements to drive technology efforts around efficient security analytics

The OpenSOC project welcomes participation from all people and organizations for development, enhancements, and/or implementation support.

What is OpenSOC?

OpenSOC is a Big Data security analytics framework designed to consume and monitor network traffic and machine exhaust data of a data center. OpenSOC is extensible and is designed to work at a massive scale.

The framework provides the following capabilities:

  • Extensible spouts and parsers for attaching OpenSOC to monitor any telemetry source
  • Extensible enrichment framework for any telemetry stream
  • Anomaly detection and real-time rules-based alerts for any telemetry stream
  • Hadoop-backed storage for telemetry stream with a customizable retention time
  • Automated real-time indexin for telemetry streams backed by Elastic Search
  • Telemetry correlation and SQL query capability for data stored in Hadoop backed by Hive
  • ODBC/JDBC compatibility and integration with existing analytics tools

OpenSOC is designed to scale up to consume millions of messages per second, enrich them, run them through anomaly detection algorithms, and issue real-time alerts.

What do I need to run OpenSOC

  • 2 Network Capture Cards (Recommend Napatech NT20E2-CAP)
  • Apache Flume 1.4.0 +
  • Apache Kafka 0.8.1+
  • Apache Storm 0.9 +
  • Apache Hadoop 2.x (any distribution)
  • Apache Hive 12 + (13 recommended)
  • Apache Hbase 0.94+
  • Elastic Search 1.1 +
  • MySQL 5.6+


OpenSOC consists of the following repositories

  • OpenSOC-Streaming:This repository contains topologies for processing, enriching, indexing, and corelating telemetry messages, PCAP reconstruction service, and various other data services. This module is open source under Apache 2.0 License. It is available on github:
  • OpenSOC-UI:UI for performing log and network packet analytics, displaying alerts, and errors. This module is open source under Apache 2.0 License. It is available on github:


Instructions for obtaining OpenSOC can be found at the project’s Primary Wiki.

Leave a Reply

Your email address will not be published. Required fields are marked *