The Exciting Frontier of
Custom KSQL Functions

Agenda

Motivation

Why are custom KSQL functions important?

Why are custom KSQL functions exciting?

KSQL functions are shareable

They facilitate exploration of the

current technological landscape

Let's explore

Agenda

Terminology

UDFs

• User-defined functions
• Operate on a single row
• Stateless

UDAFs

• User-defined aggregate functions
• Multiple inputs, one output (aggregation)
• Stateful

The process of building custom KSQL functions is

easy and repeatable

Maven Archetype

Start with the business logic

Add annotations

Deploy

Verify

Invoke

What about UDAF s ?

Build and Deploy

(same as before)

Agenda

Sentiment Analysis

• Product reception
• Outage impact
• Audience engagement
• Abusive content moderation

Natural Language API

Configs vs Environment Variables

Maximizing throughput

Use cases

Example

input sourced from user

"I would like to book a room" - user123

response generated by Dialogflow via KSQL

"I can help with that. Where would you like to reserve a room?"

hybrid training

• Pre-trained ML models
• User can also provide training data

How do we safely improve the model over time?

In event-driven architectures, this is easy

"By storing only the events and never the commands, we have a wealth of capability that supports evolutionary change" - Neil Avery

Neil Avery

Error flows

.
                    _ ._  _ , _ ._
                  (_ ' ( `  )_  .__)
                ( (  (    )   `)  ) _)
              (__ (_   (_ . _) _) ,__)
                  `~~`\ ' . /`~~`
                        ;   ;
                        /   \
          _____________/_ __ \_____________         .

• Fail fast
• Fail silently
• Dead letters

Robin

https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues

Agenda

• training models is easy
• models can be exported to Java classes

Let's see how easy it is to

build & export a model with h2o

Embed the model

Remote vs Embedded

Remote

Kai

https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning

Checkout Kai's anomaly detection UDF for another h2o example

Kai

https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning

Agenda

Installing guest languages

Graal updater (gu)

$ gu install ruby

$ gu available

ComponentId              Version             Component name
----------------------------------------------------------------
python                   1.0.0-rc15          Graal.Python
R                        1.0.0-rc15          FastR
ruby                     1.0.0-rc15          TruffleRuby

Now, let's create a Polyglot UDF!

Gotchas

• Need benchmarks. Initial tests show a start up penalty for some languages
• Using libs in guest languages may not always work
• Encountered silent and hard-to-debug failures

Possible for full integration into KSQL?

I built a Proof of Concept  (POC)

POC

• Multilingual UDFs in interactive mode
• Experimental KSQL language extensions

Inline Python UDF   POC only

The POC shows that polyglot UDFs are possible...

But inline Java UDFs may come first

Agenda

Recap

What did we learn through these examples?

Vision

We have the ingredients for a rich ecosystem

There should be a community for sharing

KSQL functions

Simply submit some info about your function at

Then others can discover your function

Now what?

Go build something exciting

Questions?