View source

== Description ==
BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.

To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.


The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].



<!--
* '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.
* '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.
* '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.
* '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.
* '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. 
* '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.
-->

== Publications ==
=== Conferences and Journals ===
*'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).<BR>Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.<BR>In the Second TPC Technology Conference on Performance Evaluation & Benchmarking (TPC TC). <BR>Collocated with VLDB10, September 17, 2010 - Singapore.
*'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).<BR>Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.<BR>In the First TPC Technology Conference on Performance Evaluation & Benchmarking (TPC TC). <BR>Collocated with VLDB09, August 24, 2009 - Lyon France.
*'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).<BR>Pedro Bizarro<BR>in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)

=== Demos ===

*'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). <BR>Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.<BR>In the Proc. of the Intl. Conf. DEBS 2008: 313-316.

=== Tutorials ===
'''Benchmarking Event Processing Systems: Current State and Future Directions'''<BR>
Marcelo Mendes, Pedro Bizarro, Paulo Marques<BR>
In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.
([[Media:BiCEP_wosp2010.pdf|pdf]])

== Software ==
*FINCoS framework, version 2.3, July 2012 ([[http://fincos.googlecode.com/files/FINCoS-2.3.zip|zip]])
The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.





<!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).<BR> Diogo Guerra, Ute Gawlick, and Pedro Bizarro.<BR>In the Proc. of the Intl. Conf. DEBS 2009.
*'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).<BR>Rafael Marmelo, Pedro Bizarro, and Paulo Marques.<BR>In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).<BR>Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.
-->