Matt Green emgeee

DataFusion Streaming Demo (2024-07-03)

At Denormalized, we've been working on adding support for streaming usecases to Apache DataFusion. This gist provides instructions for testing the current state.

Prerequisites

Docker + docker compose
Rust/Cargo installed

	use datafusion::common::test_util::datafusion_test_data;
	use datafusion::datasource::provider_as_source;
	use datafusion::error::Result;
	use datafusion::prelude::*;
	use datafusion_expr::{col, max, min, LogicalPlanBuilder};

	/// This example demonstrates executing a simple query against an Arrow data source (CSV) and
	/// fetching results with streaming aggregation and streaming window
	#[tokio::main]
	async fn main() -> Result<()> {

	### Keybase proof

	I hereby claim:

	* I am emgeee on github.
	* I am mlgreen (https://keybase.io/mlgreen) on keybase.
	* I have a public key ASBSm4Da7Kyb6JmN43y1tNq_9La1mYh8kMXRqJoEf32XUQo

	To claim this, I am signing this object:

	app.config(function ($provide) {
	$provide.decorator('Widget', function ($delegate) {
	$delegate.prototype.sing = function () {
	console.log('LA LA LA LA LA')
	}

	return $delegate
	})
	})

	var N = 50

	function asyncFunc (cb) {
	setTimeout(() => cb(Math.random()), 100)
	}

	function loop (N, results, done) {
	// Recursion base-case
	if (results.length >= N) return done(results)

	'use strict';
	/*
	* This gist is meant to accompany the blog post at
	* https://emgeee.com/making-multiple-ajax-requests-with-es6-generators
	*/

	var request = require('request')
	var Promise = require('bluebird')
	var fs = require('fs')
	var crypto = require('crypto')

	'use strict';
	var gulp = require('gulp');
	var gulpUglify = require('./index.js');
	var through = require('through2');

	var errorStream = through.obj(function(file, encoding, callback) {
	// Do something improper
	var b = a.c;

	callback();

	'use strict';

	var checkXPath = '//*[@id="gc-appbar"]/div/div[2]/div[1]/div[1]/span/div';
	var archiveXPath = '//*[@id="gc-appbar"]/div/div[2]/div[2]';

	function check() {
	var click = new MouseEvent('click', {
	'view': window,
	'bubbles': true,
	'cancelable': false

Matt Green emgeee

DataFusion Streaming Demo (2024-07-03)

Prerequisites

Configure Kafka