Octopus Logging Server

Release 0.9 rc1 out now! We're still lacking some documentation (except the inline comments), but please give it a go, if you want to. More on the News page.

The high performance logging server Octopus is intended to provide a central, (mostly) platform-indepent implementation for logging data from various sources, for example web sites, various servers, monitoring tools, and so on. That sounds pretty general, so let's go more into detail

What it does in a nutshell:

• receiving logging data from any client via HTTP
• answering with HTTP 200/OK immediately, processing the request asynchronously and in parallel (threaded)
• validating the logging input (and silently discarding invalid requests)
• storing the LogItems in an internal, in-memory LogQueue (size, thresholds, timer etc. configurable)
• another thread takes chunks of the queue and buffers them on disk (one-pass binary serialisation for maximum speed)
• yet another thread picks them up and stores them in a database
• a central panic monitor controls behaviour of all components, and supplies a status listener (another HTTP port) with run-time information, suitable for further processing by external monitoring tools

The multi-threaded processing in the first instance assures minimum response times (hence delays the clients as few as possible), whereas the multi-level persistence (memory, disk, DB) ensures no data gets lost, even if a hardware component fails.

Currently, even in a poorly performing environment, Octopus can easily process 2000 requests / second (tested on a GBit internal network, where the client, the LogServer and the database server run on different VMware ESXi virtual machines on multiple host).

In a future version, the system shall become modular, that is Octopus instances on different physical machines can talk to each other in case any back-ends fail on one of them. Plus, depending on the log data or client, data could be transmitted to different databases in the final stage.

From the database, any client can read the logged data.

The prototype is already working, and can be considered reasonably stable.

Resources and Dependencies

See the "Big Picture" to get a better overview of the different components and their interaction.

The Roadmap lists the features and changes planned for the next releases. It also includes known issues from our JIRA issue tracking

We'd like to thank the following open source projects, which we use in Octopus, for their efforts:

• C# Webserver - Lesser GNU General Public License, Version 2.1
• Apache log4net - Apache License, Version 2.0
• MySQL Connector/NET 5.2 - GNU General Public License, Version 2.0
• SQLite - Public Domain

License / Source Code

Project Maintainer: Carsten

Octopus is licensed under GNU General Public License v3.

It makes use of the aforementioned resources as-is. However, we may inherit the C# Webserver and slim it down at some point (or use the "lite" branch, which is under active development).

Public SVN Repository To browse the project's SVN repository, you can either use the ViewVC Browser or your SVN client, using this URL: http://svn.ossafe.org/repo/pub/projects

Target Platforms

The LogServer has been tested on Windows/.NET 2.0 and on Mono 2.2 (CentOS 5.2 64bit, Mono Live CD). We aim to keep this compatibility for the future as well.