http://bicep.dei.uc.pt/index.php?feed=atom&target=Marcelo&title=Special%3AContributions 2026-09-01T01:26:50Z From BiCEP MediaWiki 1.15.2 http://bicep.dei.uc.pt/index.php/Benchmarks 2014-04-24T17:30:31Z

<p>Marcelo:&#32;/* Pairs */</p> <hr /> <div>== ''Pairs'' ==<br /> The scenario for ''Pairs'' is an investment firm where a number of analysts interact with an enterprise trading system responsible for automating and optimizing the execution of orders in stock markets. Users of the system pose trading strategies which are continuously matched against live stock market data. The task of an event processing system implementing ''Pairs'' is thus to process this tick stream and compute, for each running strategy, a set of indicators, signalizing whenever those reveal an opportunity to capitalize on market inefficiencies. <br /> <br /> ''Pairs'' was designed to assess the ability of CEP systems in processing increasingly larger number of continuous queries and event arrival rates while providing quick answers – three quality attributes equally important for an event processing engine. For that, the benchmark exercises a wide range of features commonly found in most event processing applications, including:<br /> * Filtering, aggregation, and correlation of events;<br /> * Detection of event patterns and trends;<br /> * State maintenance;<br /> * Large number of simultaneous queries (increasing with the system scale);<br /> * Changing load conditions.<br /> <br /> The benchmark should be implemented as illustrated in the figure below:<br /> <br /> [[File:Benchmark Flow.png]]<br /> <br /> Initially, the user specifies a couple of workload parameters or, alternatively, uses the standard benchmark configuration to create a test setup (1). Then, a data generator application generates data and auxiliary files (2), which are used afterwards by a query generator to produce the strategies that compose the benchmark workload (3). The output of the query generator is then parsed by a vendor-specific translator, which converts the workload, initially represented in a neutral format (e.g., xml file), into the query language used by the SUT (4). After loading the query/rule set into the SUT (5), the user starts a performance run (6). During the run, the benchmark driver (FINCoS) loads the generated data file and submits the events on it to the SUT (7), which in turn returns the corresponding results to the framework (8). After test completion, a validator verifies the correctness of the answers produced by the SUT (9).<br /> <br /> All the aforementioned tools are written in Java and are available for download in the [http://bicep.dei.uc.pt/index.php/Tools Tools] section. Further details about the ''Pairs'' benchmark can be found on its [http://bicep.dei.uc.pt/images/8/8f/Pairs_Benchmark_%28rev._1.0%29.pdf specification] (currently on revision 1.0).</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-12-18T16:36:10Z

<p>Marcelo:&#32;/* What's New */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''15-Oct-2013''': “''Pairs''” benchmark updated. A new version of the benchmark specification has been released ([http://bicep.dei.uc.pt/images/8/8f/Pairs_Benchmark_%28rev._1.0%29.pdf view]).<br /> <br /> * '''17-Apr-2013''': FINCoS certified by the Standard Performance Evaluation Corporation (SPEC). The FINCoS framework has undergone a thorough review process, having been accepted to integrate SPEC Research Group’s repository of quantitative evaluation and analysis tools ([http://research.spec.org/tools.html More info]).<br /> <br /> * '''20-Mar-2013''': A new version of the FINCoS Framework (2.4.2) has been released ([http://fincos.googlecode.com/files/FINCoS%202.4.2.zip Download]).<br /> <br /> * '''06-Fev-2013''': New Publication: the paper titled “''Towards a Standard Event Processing Benchmark''” has been accepted to be presented in the Vision/Work in Progress track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor enhancements and fixes.<br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Pairs_Benchmark_(rev._1.0).pdf 2013-12-18T16:30:52Z

<p>Marcelo:&#32;Revision 1.0 of the Pairs Benchmark</p> <hr /> <div>Revision 1.0 of the Pairs Benchmark</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-09-11T17:32:58Z

<p>Marcelo:&#32;/* What's New */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''17-Apr-2013''': FINCoS certified by the Standard Performance Evaluation Corporation (SPEC). The FINCoS framework has undergone a thorough review process, having been accepted to integrate SPEC Research Group’s repository of quantitative evaluation and analysis tools ([http://research.spec.org/tools.html More info]).<br /> <br /> * '''20-Mar-2013''': A new version of the FINCoS Framework (2.4.2) has been released ([http://fincos.googlecode.com/files/FINCoS%202.4.2.zip Download]).<br /> <br /> * '''06-Fev-2013''': New Publication: the paper titled “''Towards a Standard Event Processing Benchmark''” has been accepted to be presented in the Vision/Work in Progress track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor enhancements and fixes.<br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-09-11T13:08:37Z

<p>Marcelo:&#32;/* What's New */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''17-Apr-2013''': FINCoS certified by the Standard Performance Evaluation Corporation (SPEC). The framework has undergone a thorough review process, having been accepted to integrate SPEC Research Group’s repository of quantitative evaluation and analysis tools ([http://research.spec.org/tools.html More info]).<br /> <br /> * '''20-Mar-2013''': A new version of the FINCoS Framework (2.4.2) has been released ([http://fincos.googlecode.com/files/FINCoS%202.4.2.zip Download]).<br /> <br /> * '''06-Fev-2013''': New Publication: the paper titled “''Towards a Standard Event Processing Benchmark''” has been accepted to be presented in the Vision/Work in Progress track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor enhancements and fixes.<br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-09-11T12:53:33Z

<p>Marcelo:&#32;/* Demos */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt; April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt; April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt; April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Flood: elastic streaming MapReduce'''. &lt;BR&gt;David Alves, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 4th ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2010), 113-114.<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-09-11T12:53:19Z

<p>Marcelo:&#32;/* Conferences and Journals */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt; April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt; April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Flood: elastic streaming MapReduce'''. &lt;BR&gt;David Alves, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 4th ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2010), 113-114.<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-03-20T20:20:01Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''20-Mar-2013''': A new version of the FINCoS Framework (2.4.2) has been released ([http://fincos.googlecode.com/files/FINCoS%202.4.2.zip Download]).<br /> <br /> * '''06-Fev-2013''': New Publication: the paper titled “''Towards a Standard Event Processing Benchmark''” has been accepted to be presented in the Vision/Work in Progress track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor enhancements and fixes.<br /> <br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/Tools 2013-03-20T20:19:53Z

<p>Marcelo:&#32;</p> <hr /> <div>== FINCoS ==<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of event processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled. <br /> <br /> FINCoS is benchmark-independent, so it can be used both as a component in a benchmark kit and also to carry out individual performance studies.<br /> The latest stable release (version 2.4.2, from March 2013) can be [http://fincos.googlecode.com/files/FINCoS%202.4.2.zip downloaded] from the project web [http://fincos.googlecode.com/ site].<br /> <br /> <br /> == ''Pairs'' Benchmark kit ==<br /> The ''Pairs'' benchmark kit is composed by four main applications:<br /> * '''Data Generator''': generates the benchmark simulated market data.<br /> * '''Query Generator''': produces the strategies that compose the benchmark workload.<br /> * '''Translator''': converts the workload, initially represented in a neutral format, into the query language used by the system under test (SUT). Note that translators are vendor-specific. Currently the kit contains a sample translator for the open-source CEP engine Esper.<br /> * '''Validator''': verifies the correctness of the answers produced by the SUT after a benchmark run.<br /> <br /> [http://bicep.dei.uc.pt/images/5/51/Pairs_Tools_%28beta%29.zip Download ''Pairs'' Tools (''currently in beta phase'')]<br /> <br /> In addition to the tools above, ''Pairs'' also require a benchmark driver to submit and receive events from the system under test. We recommend using the FINCoS framework to perform this function. A sample test setup file is available [http://bicep.dei.uc.pt/images/c/ca/Pairs_setup_files.zip here].</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-02-11T18:20:59Z

<p>Marcelo:&#32;/* Conferences and Journals */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Flood: elastic streaming MapReduce'''. &lt;BR&gt;David Alves, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 4th ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2010), 113-114.<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-02-11T18:20:15Z

<p>Marcelo:&#32;/* Demos */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Work in Progress and Vision track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Industry track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Flood: elastic streaming MapReduce'''. &lt;BR&gt;David Alves, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 4th ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2010), 113-114.<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-02-11T18:16:35Z

<p>Marcelo:&#32;/* Tutorials */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Work in Progress and Vision track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Industry track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Posters and Demonstrations Track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Flood: elastic streaming MapReduce'''. &lt;BR&gt;David Alves, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 4th ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2010), 113-114.<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-02-11T18:10:39Z

<p>Marcelo:&#32;/* Demos */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Work in Progress and Vision track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Industry track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Posters and Demonstrations Track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Flood: elastic streaming MapReduce'''. &lt;BR&gt;David Alves, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 4th ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2010), 113-114.<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-02-11T18:06:15Z

<p>Marcelo:&#32;/* Conferences and Journals */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Towards a Standard Event Processing Benchmark'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Work in Progress and Vision track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Industry track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Posters and Demonstrations Track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-02-11T18:00:40Z

<p>Marcelo:&#32;/* Demos */</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Industry track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> *'''FINCoS: Benchmark Tools for Event Processing Systems'''. &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Posters and Demonstrations Track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Benchmarks 2013-01-24T19:27:40Z

<p>Marcelo:&#32;</p> <hr /> <div>== ''Pairs'' ==<br /> The scenario for ''Pairs'' is an investment firm where a number of analysts interact with an enterprise trading system responsible for automating and optimizing the execution of orders in stock markets. Users of the system pose trading strategies which are continuously matched against live stock market data. The task of an event processing system implementing ''Pairs'' is thus to process this tick stream and compute, for each running strategy, a set of indicators, signalizing whenever those reveal an opportunity to capitalize on market inefficiencies. <br /> <br /> ''Pairs'' was designed to assess the ability of CEP systems in processing increasingly larger number of continuous queries and event arrival rates while providing quick answers – three quality attributes equally important for an event processing engine. For that, the benchmark exercises a wide range of features commonly found in most event processing applications, including:<br /> * Filtering, aggregation, and correlation of events;<br /> * Detection of event patterns and trends;<br /> * State maintenance;<br /> * Large number of simultaneous queries (increasing with the system scale);<br /> * Changing load conditions.<br /> <br /> The benchmark should be implemented as illustrated in the figure below:<br /> <br /> [[File:Benchmark Flow.png]]<br /> <br /> Initially, the user specifies a couple of workload parameters or, alternatively, uses the standard benchmark configuration to create a test setup (1). Then, a data generator application generates data and auxiliary files (2), which are used afterwards by a query generator to produce the strategies that compose the benchmark workload (3). The output of the query generator is then parsed by a vendor-specific translator, which converts the workload, initially represented in a neutral format (e.g., xml file), into the query language used by the SUT (4). After loading the query/rule set into the SUT (5), the user starts a performance run (6). During the run, the benchmark driver (FINCoS) loads the generated data file and submits the events on it to the SUT (7), which in turn returns the corresponding results to the framework (8). After test completion, a validator verifies the correctness of the answers produced by the SUT (9).<br /> <br /> All the aforementioned tools are written in Java and are available for download in the [http://bicep.dei.uc.pt/index.php/Tools Tools] section. Further details about the ''Pairs'' benchmark can be found on its [http://bicep.dei.uc.pt/images/b/ba/Pairs_Benchmark_%28rev._0.2%29.pdf specification] (currently on revision 0.2).</div>

Marcelo http://bicep.dei.uc.pt/index.php/Tools 2013-01-24T19:23:47Z

<p>Marcelo:&#32;</p> <hr /> <div>== FINCoS ==<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of event processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled. <br /> <br /> FINCoS is benchmark-independent, so it can be used both as a component in a benchmark kit and also to carry out individual performance studies.<br /> The latest stable release (version 2.4.1, from December 2012) can be [http://fincos.googlecode.com/files/FINCoS%202.4.1.zip downloaded] from the project web [http://fincos.googlecode.com/ site].<br /> <br /> <br /> == ''Pairs'' Benchmark kit ==<br /> The ''Pairs'' benchmark kit is composed by four main applications:<br /> * '''Data Generator''': generates the benchmark simulated market data.<br /> * '''Query Generator''': produces the strategies that compose the benchmark workload.<br /> * '''Translator''': converts the workload, initially represented in a neutral format, into the query language used by the system under test (SUT). Note that translators are vendor-specific. Currently the kit contains a sample translator for the open-source CEP engine Esper.<br /> * '''Validator''': verifies the correctness of the answers produced by the SUT after a benchmark run.<br /> <br /> [http://bicep.dei.uc.pt/images/5/51/Pairs_Tools_%28beta%29.zip Download ''Pairs'' Tools (''currently in beta phase'')]<br /> <br /> In addition to the tools above, ''Pairs'' also require a benchmark driver to submit and receive events from the system under test. We recommend using the FINCoS framework to perform this function. A sample test setup file is available [http://bicep.dei.uc.pt/images/c/ca/Pairs_setup_files.zip here].</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Benchmark_Flow.png 2013-01-24T19:16:54Z

<p>Marcelo:&#32;uploaded a new version of &quot;File:Benchmark Flow.png&quot;:&amp;#32;smaller</p> <hr /> <div>Pairs benchmark execution</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Benchmark_Flow.png 2013-01-24T19:11:46Z

<p>Marcelo:&#32;uploaded a new version of &quot;File:Benchmark Flow.png&quot;:&amp;#32;Smaller size</p> <hr /> <div>Pairs benchmark execution</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Benchmark_Flow.png 2013-01-24T19:10:05Z

<p>Marcelo:&#32;Pairs benchmark execution</p> <hr /> <div>Pairs benchmark execution</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-01-21T23:57:30Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor enhancements and fixes.<br /> <br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/Tools 2013-01-21T23:51:04Z

<p>Marcelo:&#32;</p> <hr /> <div>== FINCoS ==<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of event processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled. <br /> <br /> FINCoS is benchmark-independent, so it can be used both as a component in a benchmark kit and also to carry out individual performance studies.<br /> The latest stable release (version 2.4.1, from December 2012) of FINCoS can be [http://fincos.googlecode.com/files/FINCoS%202.4.1.zip downloaded] from the project web [http://fincos.googlecode.com/ site].<br /> <br /> <br /> == ''Pairs'' Benchmark kit ==<br /> The ''Pairs'' benchmark kit is composed by four main applications:<br /> * '''Data Generator''': generates the benchmark simulated market data.<br /> * '''Query Generator''': produces the strategies that compose the benchmark workload.<br /> * '''Translator''': converts the workload, initially represented in a neutral format, into the query language used by the system under test (SUT). Note that translators are vendor-specific. Currently the kit contains a sample translator for the open-source CEP engine Esper.<br /> * '''Validator''': verifies the correctness of the answers produced by the SUT after a benchmark run.<br /> <br /> [http://bicep.dei.uc.pt/images/5/51/Pairs_Tools_%28beta%29.zip Download ''Pairs'' Tools (''currently in beta phase'')]<br /> <br /> In addition to the tools above, ''Pairs'' also require a benchmark driver to submit and receive events from the system under test. We recommend using the FINCoS framework to perform this function. A sample test setup file is available [http://bicep.dei.uc.pt/images/c/ca/Pairs_setup_files.zip here].</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Pairs_setup_files.zip 2013-01-21T23:50:30Z

<p>Marcelo:&#32;Setup file used to run the Pairs benchmark using FINCoS</p> <hr /> <div>Setup file used to run the Pairs benchmark using FINCoS</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Pairs_Tools_(beta).zip 2013-01-21T23:39:18Z

<p>Marcelo:&#32;uploaded a new version of &quot;File:Pairs Tools (beta).zip&quot;</p> <hr /> <div>Beta release of the Pairs benchmark kit</div>

Marcelo http://bicep.dei.uc.pt/index.php/Benchmarks 2013-01-21T23:06:33Z

<p>Marcelo:&#32;</p> <hr /> <div><br /> == ''Pairs'' ==<br /> The scenario for ''Pairs'' is an investment firm where a number of analysts interact with an enterprise trading system responsible for automating and optimizing the execution of orders in stock markets. Users of the system pose trading strategies which are continuously matched against live stock market data. The task of an event processing system implementing ''Pairs'' is thus to process this tick stream and compute, for each running strategy, a set of indicators, signalizing whenever those reveal an opportunity to capitalize on market inefficiencies. <br /> <br /> ''Pairs'' was designed to assess the ability of CEP systems in processing increasingly larger number of continuous queries and event arrival rates while providing quick answers – three quality attributes equally important for an event processing engine. For that, the benchmark exercises a wide range of features commonly found in most event processing applications, including:<br /> * Filtering, aggregation, and correlation of events;<br /> * Detection of event patterns and trends;<br /> * State maintenance;<br /> * Large number of simultaneous queries (increasing with the system scale);<br /> * Changing load conditions.<br /> <br /> The most updated specification of ''Pairs'' (revision 0.2) can be found [http://bicep.dei.uc.pt/images/b/ba/Pairs_Benchmark_%28rev._0.2%29.pdf here]. The benchmark kit, required to run ''Pairs'', can be donwloaded from the [http://bicep.dei.uc.pt/index.php/Tools Tools] section.</div>

Marcelo http://bicep.dei.uc.pt/index.php/Tools 2013-01-21T23:03:32Z

<p>Marcelo:&#32;</p> <hr /> <div>== FINCoS ==<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of event processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled. <br /> <br /> FINCoS is benchmark-independent, so it can be used both as a component in a benchmark kit and also to carry out individual performance studies.<br /> The latest stable release (version 2.4.1, from December 2012) of FINCoS can be [http://fincos.googlecode.com/files/FINCoS%202.4.1.zip downloaded] from the project web [http://fincos.googlecode.com/ site].<br /> <br /> <br /> == ''Pairs'' Benchmark kit ==<br /> The ''Pairs'' benchmark kit is composed by four main applications:<br /> * '''Data Generator''': generates the benchmark simulated market data.<br /> * '''Query Generator''': produces the strategies that compose the benchmark workload.<br /> * '''Translator''': converts the workload, initially represented in a neutral format, into the query language used by the system under test (SUT). Note that translators are vendor-specific. Currently the kit contains a sample translator for the open-source CEP engine Esper.<br /> * '''Validator''': verifies the correctness of the answers produced by the SUT after a benchmark run.<br /> <br /> [http://bicep.dei.uc.pt/images/5/51/Pairs_Tools_%28beta%29.zip Download ''Pairs'' Tools (''currently in beta phase'')]<br /> <br /> In addition to the tools above, ''Pairs'' also require a benchmark driver to submit and receive events from the system under test. We recommend using the FINCoS framework to perform this function. A sample test setup file is available [http://bicep.dei.uc.pt here].</div>

Marcelo http://bicep.dei.uc.pt/index.php/Benchmarks 2013-01-21T23:02:11Z

<p>Marcelo:&#32;</p> <hr /> <div>In this page you can find a description of the benchmarks that comprise the ''BiCEP'' suite.<br /> <br /> == ''Pairs'' ==<br /> The scenario for ''Pairs'' is an investment firm where a number of analysts interact with an enterprise trading system responsible for automating and optimizing the execution of orders in stock markets. Users of the system pose trading strategies which are continuously matched against live stock market data. The task of an event processing system implementing ''Pairs'' is thus to process this tick stream and compute, for each running strategy, a set of indicators, signalizing whenever those reveal an opportunity to capitalize on market inefficiencies. <br /> <br /> ''Pairs'' was designed to assess the ability of CEP systems in processing increasingly larger number of continuous queries and event arrival rates while providing quick answers – three quality attributes equally important for an event processing engine. For that, the benchmark exercises a wide range of features commonly found in most event processing applications, including:<br /> * Filtering, aggregation, and correlation of events;<br /> * Detection of event patterns and trends;<br /> * State maintenance;<br /> * Large number of simultaneous queries (increasing with the system scale);<br /> * Changing load conditions.<br /> <br /> The most updated specification of ''Pairs'' (revision 0.2) can be found [http://bicep.dei.uc.pt/images/b/ba/Pairs_Benchmark_%28rev._0.2%29.pdf here]. The benchmark kit, required to run ''Pairs'', can be donwloaded from the [http://bicep.dei.uc.pt/Tools Tools] section.</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Pairs_Tools_(beta).zip 2013-01-21T23:01:45Z

<p>Marcelo:&#32;Beta release of the Pairs benchmark kit</p> <hr /> <div>Beta release of the Pairs benchmark kit</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:Pairs_Benchmark_(rev._0.2).pdf 2013-01-21T23:00:14Z

<p>Marcelo:&#32;Revision 0.2 of the Pairs Benchmark</p> <hr /> <div>Revision 0.2 of the Pairs Benchmark</div>

Marcelo http://bicep.dei.uc.pt/index.php/MediaWiki:Sidebar 2013-01-21T22:57:49Z

<p>Marcelo:&#32;</p> <hr /> <div>* navigation<br /> ** mainpage|mainpage<br /> ** benchmarks|Benchmarks<br /> ** tools|Tools<br /> ** publications|Publications<br /> ** people|People</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-01-21T22:56:03Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor improvements and fixes.<br /> <br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-01-21T22:55:00Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == What's New ==<br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor improvements and fixes.<br /> <br /> <br /> == ==</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2013-01-21T22:51:56Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). The project, partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant], started in September 2007. Since then, we have developed benchmarking tools, carried out a number of performance evaluations on real CEP engines, and studied several use-cases of the technology. Currently, we are working towards the definition of novel application benchmarks that allow assessing the overall performance and scalability of event processing platforms. <br /> <br /> The ''BiCEP benchmark suite'' shall comprise a number of smaller, domain-specific benchmarks, each with its own workload, dataset and metrics. Our goal is that each benchmark will allow evaluating one or more aspects of event processing systems (e.g. latency, scalability with respect to number of queries and rules, storage efficiency, etc.). <br /> <br /> <br /> &lt;!--<br /> In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> == What's New ==<br /> * '''21-Jan-2013''': Benchmark “''Pairs''” released. The first of the BiCEP domain-specific benchmarks is now available. The benchmark description and its specification can be found [http://bicep.dei.uc.pt/index.php/Benchmarks here].<br /> <br /> * '''09-Jan-2013''': New Publication: the paper titled “''Overcoming Memory Limitations in High-Throughput Event-Based Applications''” has been accepted to be presented in the Industry track of 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), to be held in Prague, from April 21 to 24. ([http://bicep.dei.uc.pt/index.php/Publications View full list of publications.])<br /> <br /> * '''10-Dec-2012''': A new version of the FINCoS Framework (2.4.1) is now available. This release fixes problems on RMI communication when running under Java 7 and brings several other minor improvements.([http://fincos.googlecode.com/files/FINCoS%202.4.zip Download])<br /> <br /> * '''07-Out-2012''': A new version of the FINCoS Framework (2.4) has been released. This version adds support for improved load generation through user-provided data files, and brings a number of other minor improvements and fixes.</div>

Marcelo http://bicep.dei.uc.pt/index.php/Tools 2013-01-21T22:50:32Z

<p>Marcelo:&#32;Created page with '== FINCoS == The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of event processing systems. It provides a flexible and …'</p> <hr /> <div>== FINCoS ==<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of event processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled. <br /> <br /> FINCoS is benchmark-independent, so it can be used both as a component in a benchmark kit and also to carry out individual performance studies.<br /> The latest stable release (version 2.4.1, from December 2012) of FINCoS can be [http://fincos.googlecode.com/files/FINCoS%202.4.1.zip downloaded] from the project web [http://fincos.googlecode.com/ site].<br /> <br /> <br /> == ''Pairs'' Benchmark kit ==<br /> The ''Pairs'' benchmark kit is composed by four main applications:<br /> * '''Data Generator''': generates the benchmark simulated market data.<br /> * '''Query Generator''': produces the strategies that compose the benchmark workload.<br /> * '''Translator''': converts the workload, initially represented in a neutral format, into the query language used by the system under test (SUT). Note that translators are vendor-specific. Currently the kit contains a sample translator for the open-source CEP engine Esper.<br /> * '''Validator''': verifies the correctness of the answers produced by the SUT after a benchmark run.<br /> <br /> [http://bicep.dei.uc.pt Download ''Pairs'' Tools (''currently in beta phase'')]<br /> <br /> In addition to the tools above, ''Pairs'' also require a benchmark driver to submit and receive events from the system under test. We recommend using the FINCoS framework to perform this function. A sample test setup file is available [http://bicep.dei.uc.pt here].</div>

Marcelo http://bicep.dei.uc.pt/index.php/Benchmarks 2013-01-21T22:21:45Z

<p>Marcelo:&#32;</p> <hr /> <div>In this page you can find a description of the benchmarks that comprise the ''BiCEP'' suite.<br /> <br /> == ''Pairs'' ==<br /> The scenario for ''Pairs'' is an investment firm where a number of analysts interact with an enterprise trading system responsible for automating and optimizing the execution of orders in stock markets. Users of the system pose trading strategies which are continuously matched against live stock market data. The task of an event processing system implementing ''Pairs'' is thus to process this tick stream and compute, for each running strategy, a set of indicators, signalizing whenever those reveal an opportunity to capitalize on market inefficiencies. <br /> <br /> ''Pairs'' was designed to assess the ability of CEP systems in processing increasingly larger number of continuous queries and event arrival rates while providing quick answers – three quality attributes equally important for an event processing engine. For that, the benchmark exercises a wide range of features commonly found in most event processing applications, including:<br /> * Filtering, aggregation, and correlation of events;<br /> * Detection of event patterns and trends;<br /> * State maintenance;<br /> * Large number of simultaneous queries (increasing with the system scale);<br /> * Changing load conditions.<br /> <br /> The most updated specification of ''Pairs'' (revision 0.2) can be found [http://bicep.dei.uc.pt here]. The benchmark kit, required to run ''Pairs'', can be donwloaded from the [http://bicep.dei.uc.pt/Tools Tools] section.</div>

Marcelo http://bicep.dei.uc.pt/index.php/MediaWiki:Sidebar 2013-01-21T21:47:34Z

<p>Marcelo:&#32;</p> <hr /> <div>* navigation<br /> ** mainpage|mainpage<br /> ** tools|Tools<br /> ** benchmarks|Benchmarks<br /> ** publications|Publications<br /> ** people|People</div>

Marcelo http://bicep.dei.uc.pt/index.php/Benchmarks 2013-01-21T21:42:05Z

<p>Marcelo:&#32;Created page with 'In this page you can find a description of the benchmarks that comprise the ''BiCEP'' suite. == ''Pairs'' == The ''Pairs'' benchmark is set on the capital markets environment an…'</p> <hr /> <div>In this page you can find a description of the benchmarks that comprise the ''BiCEP'' suite.<br /> <br /> == ''Pairs'' ==<br /> The ''Pairs'' benchmark is set on the capital markets environment and exercises a wide range of features commonly found in most event processing applications, including:<br /> * Filtering, aggregation, and correlation of events;<br /> * Detection of event patterns and trends;<br /> * State maintenance;<br /> * Large number of simultaneous queries (increasing with the system scale);<br /> * Changing load conditions.<br /> <br /> ''Pairs'' was designed to assess the ability of CEP systems in processing increasingly larger number of continuous queries and event arrival rates while providing quick answers – three quality attributes equally important for an event processing engine. The benchmark was also designed to be fully customizable, so that users can carry out performance studies that resemble more closely their own environments.<br /> <br /> The most updated specification of ''Pairs'' (revision 0.2) can be found [http://bicep.dei.uc.pt here]. The benchmark kit, required to run ''Pairs'', can be donwloaded from the [http://bicep.dei.uc.pt/Tools Tools] section.</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-01-21T19:47:43Z

<p>Marcelo:&#32;</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Overcoming Memory Limitations in High-Throughput Event-Based Applications'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013).&lt;BR&gt;(''Industry track, to appear'') April, 2013 - Prague, Czech Republic.<br /> <br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems'''.&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> <br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> <br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> <br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/MediaWiki:Sidebar 2013-01-21T19:15:06Z

<p>Marcelo:&#32;</p> <hr /> <div>* navigation<br /> ** mainpage|mainpage<br /> ** publications|Publications<br /> ** people|People</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-01-21T19:13:42Z

<p>Marcelo:&#32;</p> <hr /> <div>In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> == Conferences and Journals ==<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''An Integrated Data Management Approach to Manage Health Care Data''' ([[Media:ICU_debs2009.pdf|pdf]])&lt;BR&gt;Diogo Guerra, Dieter Gawlick, and Pedro Bizarro.&lt;BR&gt;In Proceedings of the 3rd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2009).<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> == Demos ==<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In Proceedings of the 2nd ACM Intl. Conf. on Distributed Event-Based Systems (DEBS 2008), 313-316.<br /> <br /> == Tutorials ==<br /> *'''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Publications 2013-01-21T19:01:29Z

<p>Marcelo:&#32;Created page with '== Publications == In this page you can find a list of publications produced in the ambit of the BiCEP project. === Conferences and Journals === *'''Assessing and Optimizing Micr…'</p> <hr /> <div>== Publications ==<br /> In this page you can find a list of publications produced in the ambit of the BiCEP project.<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2012-12-10T22:04:28Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> <br /> The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> <br /> <br /> &lt;!--<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.4.1, December 2012 ([http://fincos.googlecode.com/files/FINCoS%202.4.1.zip download])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.<br /> <br /> <br /> <br /> <br /> <br /> &lt;!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2012-10-08T01:32:30Z

<p>Marcelo:&#32;</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> <br /> The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> <br /> <br /> &lt;!--<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.4, October 2012 ([http://fincos.googlecode.com/files/FINCoS%202.4.zip download])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.<br /> <br /> <br /> <br /> <br /> <br /> &lt;!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2012-07-15T15:40:10Z

<p>Marcelo:&#32;/* Software */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> <br /> The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> <br /> <br /> &lt;!--<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.3, July 2012 ([http://fincos.googlecode.com/files/FINCoS-2.3.zip download])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.<br /> <br /> <br /> <br /> <br /> <br /> &lt;!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2012-07-15T15:37:30Z

<p>Marcelo:&#32;/* Software */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> <br /> The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> <br /> <br /> &lt;!--<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.3, July 2012 ([http://fincos.googlecode.com/files/FINCoS-2.3.zip] download)<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.<br /> <br /> <br /> <br /> <br /> <br /> &lt;!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2012-07-15T15:35:06Z

<p>Marcelo:&#32;/* Software */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> <br /> The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> <br /> <br /> &lt;!--<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.3, July 2012 ([[http://fincos.googlecode.com/files/FINCoS-2.3.zip|zip]])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.<br /> <br /> <br /> <br /> <br /> <br /> &lt;!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2012-07-15T15:34:03Z

<p>Marcelo:&#32;/* Software */</p> <hr /> <div>== Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). In the papers below we present benchmarking experiments that evaluate engines CEP along several metrics. Some of these are obvious, others not quite: throughput, maximum latency, latency degradation ratio, post-peak latency variation ratio, memory consumption, instruction count, clocks-per-instruction, cache miss rates and more. We studied many different basic and not so basic operations such as selection, projections, aggregations, joins, windows, and patterns detection and the impact of types and sizes of windows, query sharing, variations to peaks, the effects of garbage collection options, and internal data structures and tuple representations.<br /> <br /> To perform all this, we built FINCoS, a testing framework (available below) that lets users generate and/or load test data, start and play benchmarking experiments, visualize and capture metrics, and run multiple engines at the same time.<br /> <br /> <br /> The project started in September 2007 and is partially funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> <br /> <br /> &lt;!--<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events.<br /> --&gt;<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''Assessing and Optimizing Microarchitectural Performance of Event Processing Systems''' ([[Media:Mendes_tpctc2010.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Second TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB10, September 17, 2010 - Singapore.<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> <br /> === Demos ===<br /> <br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> === Tutorials ===<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.2.1, December 2009 ([[http://fincos.googlecode.com/files/FINCoS-2.3.zip|zip]])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.<br /> <br /> <br /> <br /> <br /> <br /> &lt;!--*'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2010-11-02T17:46:58Z

<p>Marcelo:&#32;Undo revision 272 by Marcelo (Talk)</p> <hr /> <div>== News ==<br /> *'''NEW tutorial: '''&lt;BR&gt;<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> &lt;!--*'''NEW paper: '''&lt;BR&gt;'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''NEW paper: '''&lt;BR&gt;'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;<br /> <br /> == Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). CEP metrics should include the following metrics (some obvious, others not quite):<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events. <br /> <br /> <br /> The project started in September 2007 and is funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> === Demos ===<br /> *'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.2.1, December 2009 ([[Media:FINCoS-2.2.1.zip|zip]])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2010-11-02T17:46:24Z

<p>Marcelo:&#32;</p> <hr /> <div>== News ==<br /> *'''NEW tutorial: '''&lt;BR&gt;<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> &lt;!--*'''NEW paper: '''&lt;BR&gt;'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''NEW paper: '''&lt;BR&gt;'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;<br /> <br /> == Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). CEP metrics should include the following metrics (some obvious, others not quite):<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events. <br /> <br /> <br /> The project started in September 2007 and is funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> === Demos ===<br /> *'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.2.1, December 2009 ([[Media:http://bicep.dei.uc.pt/images/0/0e/FINCoS-2.2.1.zip|zip]])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.</div>

Marcelo http://bicep.dei.uc.pt/index.php/File:FINCoS-2.2.1.zip 2010-11-02T17:41:50Z

<p>Marcelo:&#32;uploaded a new version of &quot;File:FINCoS-2.2.1.zip&quot;:&amp;#32;(updated user guide)</p> <hr /> <div>FINCoS version 2.2, December 2009</div>

Marcelo http://bicep.dei.uc.pt/index.php/Main_Page 2010-10-31T13:30:59Z

<p>Marcelo:&#32;</p> <hr /> <div>== News ==<br /> *'''NEW tutorial: '''&lt;BR&gt;<br /> '''Benchmarking Event Processing Systems: Current State and Future Directions'''&lt;BR&gt;<br /> Marcelo Mendes, Pedro Bizarro, Paulo Marques&lt;BR&gt;<br /> In the First Joint WOSP/SIPEW International Conference on Performance Engineering, San Jose, California, USA, January 28, 2010.<br /> ([[Media:BiCEP_wosp2010.pdf|pdf]])<br /> <br /> &lt;!--*'''NEW paper: '''&lt;BR&gt;'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''NEW paper: '''&lt;BR&gt;'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> --&gt;<br /> <br /> == Description ==<br /> BiCEP, a project from the Systems and Software Engineering Group at the University of Coimbra, has the goal to study and improve performance and produce benchmarks for [http://www.complexevents.com:Complex Event Processing] systems (CEP). CEP metrics should include the following metrics (some obvious, others not quite):<br /> * '''Sustainable throughput''': the steady-state number of events per unit of time that a (warmed-up) CEP engine can process while performing query processing. Even within the same system, sustainable throughput can vary widely depending on the amount of work to be done during query processing.<br /> * '''Response time''': the time since the last event of some event pattern is fed into the system until the system notifies the event pattern detection.<br /> * '''Scalability''': Unlike other benchmarks that considerer scalability only as a variation of the benchmark with more data and more users, in BiCEP we would like scalability to be a first-class metric. That is, while it is useful to compare systems at different scale levels, it is also very interesting to assess how well a given system scales. For example, CEP engines can use some of the new techniques (e.g., [http://aws.amazon.com/ec2 Amazon Elastic Compute Cloud]) that allow a system to grab hardware resources on demand makes. We are planning scalability experiments along three directions: i) scale-up: increase the system and increase the load, ii) speed-up: increase the system and maintain the load, and iii) load-up: maintain system but increase the load.<br /> * '''Adaptivity''': Typically, systems are benchmarked after they are “warmed-up” and in a steady state. However, while it seems that there will be periods where CEP systems are in steady states, it also appears likely that, due to the very unpredictable nature of the real-world events being processed by CEP engines, there will be frequent disruptive moments, when the system should adapt its query processing to be more efficient.<br /> * '''Computation Sharing''': Many CEP applications process tens, hundreds, millions of similar queries concurrently. For example, a CEP engine in a financial trading company may be processing thousands of rules for each stock ticket: many customers may be monitoring the same stock but each customer may have slightly different buy or sell values. If the CEP engine can devise query processing techniques such that different queries are able to share computation, then the scalability potential of the system is greatly improved. <br /> * '''Similarity search and precision and recall''': As far as we know, no CEP engine uses any kind of similarity search: the patterns being searched are always precisely specified by a query language. Thus, we expect no false positives and no false negatives. However, if CEP users demand more and more complex patterns, we expect CEP engines to start using similarity search. If similarity search is used, then CEP engines may occasionally produce incorrect results by way of false positives and false negatives. We also expect false positives and false negatives if CEP engines use past events to forecast real-world future events. <br /> <br /> <br /> The project started in September 2007 and is funded by an [http://cordis.europa.eu/mariecurie-actions/irg/home.html FP6 Marie Curie International Reintegration Grant].<br /> <br /> == Publications ==<br /> === Conferences and Journals ===<br /> *'''A Performance Study of Event Processing Systems''' ([[Media:Mendes_tpctc2009.pdf|pdf]]).&lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First TPC Technology Conference on Performance Evaluation &amp; Benchmarking (TPC TC). &lt;BR&gt;Collocated with VLDB09, August 24, 2009 - Lyon France.<br /> *'''9ticks – The Web as a Stream''' ([[Media:9ticks_dait2009.pdf‎|pdf]]).&lt;BR&gt;Rafael Marmelo, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the First International Workshop on Database Architectures for the Internet of Things (DAIT 2009).&lt;BR&gt;Extended version ([[Media:9ticks_iete2009.pdf|pdf]]) also published on the 2009 September/October issue of IETE Technical Review journal.<br /> *'''BiCEP - Benchmarking Complex Event Processing Systems''' ([http://drops.dagstuhl.de/opus/volltexte/2007/1143/pdf/07191.BizarroPedro.ExtAbstract.1143.pdf pdf]).&lt;BR&gt;Pedro Bizarro&lt;BR&gt;in [http://www.dagstuhl.de Dagshtul] Event Processing Seminar, 2007. (POSITION PAPER)<br /> === Demos ===<br /> *'''An Integrated Data Management Approach to Manage Health Care Sensor Data''' ([[Media:ICU_debs2009.pdf|pdf]]).&lt;BR&gt; Diogo Guerra, Ute Gawlick, and Pedro Bizarro.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2009.<br /> *'''A framework for performance evaluation of complex event processing systems''' ([[Media:FINCoS_DEBS2008.pdf|pdf]]). &lt;BR&gt;Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques.&lt;BR&gt;In the Proc. of the Intl. Conf. DEBS 2008: 313-316.<br /> <br /> == Software ==<br /> *FINCoS framework, version 2.2.1, December 2009 ([[Media:FINCoS-2.2.1.zip|zip]])<br /> The FINCoS framework is a Java-based set of benchmarking tools for load generation and performance measuring of Event Processing systems. It provides a flexible and neutral approach for experimenting diverse CEP systems, where load generators, datasets, queries, and adapters can be easily attached, swapped, reconfigured and scaled.</div>

Marcelo