173 lines
17 KiB
HTML
173 lines
17 KiB
HTML
<!DOCTYPE html>
|
|
<html><head><meta charset="utf-8"><meta content="width=device-width, initial-scale=1" name="viewport"><link href="/reveal/css/reveal.css" rel="stylesheet" type="text/css"><link href="/font-awesome-4.7.0/css/font-awesome.css" rel="stylesheet" type="text/css"><link href="/highlight/styles/darcula.css" rel="stylesheet" type="text/css"><link href="/reveal/css/theme/night.css" rel="stylesheet" type="text/css"></head><body><div class="reveal"><div class="slides"><section><h1>Transducers: a deep dive</h1></section><section><h3>Road map <i aria-hidden="true" class="fa fa-map"></i></h3><ul><li>What are transducers</li><li>Use and create transducers</li><li>A review of map function</li></ul></section><section><section><h3>What is a Transducer?</h3></section><section data-markdown><script type="text/template">#### According to Wikipedia
|
|
> A **transducer** is a device that converts one form of energy to another
|
|
> ...
|
|
>Transducers are often employed at the boundaries...</script></section><section data-markdown><script type="text/template">#### According to clojuredoc
|
|
A **transducer** (sometimes referred to as **xform** or **xf**) is a
|
|
transformation from one reducing function to another</script></section><section data-markdown><script type="text/template">#### According to Rich Hickey
|
|
* extract the **essence** of map, filter et al
|
|
* away of the functions that transform sequences/collections
|
|
* so they can be use elsewhere
|
|
* recasting them as **process transformation**</script></section><section data-markdown><script type="text/template">#### That does not help much (the first time)
|
|
Instead let take a look at what transducer try to solve</script></section><section><div data-markdown><script type="text/template">#### let's look at an example
|
|
We have two options...
|
|
1. look at code example
|
|
2. look at a very nice analogy</script></div><i aria-hidden="true" class="fa fa-question"></i></section><section><div data-markdown><script type="text/template">#### Classical problem: $ \sum_{n=0}^{N-1} 2 \cdot n + 3 $</script></div><pre><code class="clojure">(defn classic-sum [N]
|
|
(let [all-numbers (range)
|
|
input (take N all-numbers)
|
|
times-two (map #(* 2 %) input)
|
|
times-two-plus-three (map #(+ 3 %) times-two)]
|
|
(reduce + 0 times-two-plus-three)))</code></pre><div data-markdown><script type="text/template">note: Here we produce many lists when we actually do not need any</script></div></section><section><div data-markdown><script type="text/template">#### And the solution to a very similar problem requires a re-written</script></div><pre><code class="clojure">(require '[clojure.core.async :as async])</code></pre><pre><code class="clojure">(defn classic-sum-channel [N]
|
|
(let [all-numbers (async/to-chan (range))
|
|
input (async/take N all-numbers)
|
|
times-two (async/map< #(* 2 %) input)
|
|
times-two-plus-three (async/map< #(+ 3 %) times-two)]
|
|
(async/reduce + 0 times-two-plus-three)))</code></pre></section><section><div data-markdown><script type="text/template">#### Transducers simplify all the above fiddling</script></div><pre><code class="clojure">(require '[clojure.core.async :as async])</code></pre><pre><code class="clojure">(defn xform [N] ;; transducers are usually named xform or xf
|
|
(comp (take N)
|
|
(map #(* 2 %))
|
|
(map #(+ 3 %))))
|
|
|
|
(defn trans-classic-sum [N]
|
|
(transduce (xform N) + 0 (range)))
|
|
|
|
(defn trans-classic-sum-channel [N]
|
|
(async/transduce (xform N) + 0 (async/to-chan (range))))</code></pre></section><section data-markdown><script type="text/template">Hence, a **transducer** is a data pipeline "functional interface"
|
|
that is un aware of the underlying data structure.</script></section></section><section><section data-markdown><script type="text/template">## Rich Hickey's analogy
|
|
> talk: https://www.youtube.com/watch?v=6mTbuzafcII</script></section><section data-markdown><script type="text/template">### To load an airplane you need to...</script></section><section><h4>1 : You start with some luggage crates</h4><img src="/images/crates.jpg"></section><section><h4>2 : unpack the luggaes from the crates</h4><img src="/images/unload.jpg"></section><section><h4>3 : remove bags that contain laptop</h4><img src="/images/laptop-bag.jpg"></section><section><h4>4 : label heavy bags</h4><img src="/images/heavy-bag.jpeg"></section><section><h4>5 : load the airplane</h4><img src="/images/load-airplane.jpg"></section><section data-markdown><script type="text/template">### Recap the analogy
|
|
| Physical action/thing | Function/data structure |
|
|
|:--------------------------------|:------------------------|
|
|
|crates of bags | seq of seq of Map |
|
|
|un-pack the crates | `mapcat` |
|
|
|remove bags that contains laptop | `filtering` |
|
|
|label heavy bags | `map` |
|
|
|airplane | sequence |</script></section><section><h3>Lessons from the analogy</h3></section><section data-markdown><script type="text/template">#### Lesson number 1
|
|
> The origin and destination of the bags do not matter
|
|
* receiving the crates in a truck or on a conveyor belt is the same
|
|
* Loading a ship or an airplane is the same thing</script></section><section data-markdown><script type="text/template">#### Lesson number 2
|
|
> There is no need to place the bags on a conveyor belt between each step</script></section></section><section><section><h3>How do we create transducers?</h3><h3>How do we use transducers?</h3></section><section data-markdown><script type="text/template">#### How to create transducers?
|
|
|
|
The majority of data processing method implement an arity that returns
|
|
a transducer. The following functions produce a transducer when the
|
|
input collection is omitted:
|
|
|
|
| | | | |
|
|
|--|--|--|--|
|
|
|map|cat|mapcat|filter|
|
|
|remove|take|take-while|take-nth|
|
|
|drop|drop-while|replace|partition-by|
|
|
|partition-all|keep|keep-indexed|map-indexed|
|
|
|distinct|interpose|dedupe|random-sample|
|
|
</script></section><section data-markdown><script type="text/template">#### How to use a transducer?
|
|
|
|
* **transduce** is can be use like reduce
|
|
* **eduction** is used to create reducible/iterable
|
|
* **into** is used to append data to an existing collection
|
|
* **sequence** can create a sequence from transducer</script></section><section><div data-markdown><script type="text/template">#### example 1</script></div><pre><code class="clojure">(def xform (map #(* 2 %)))
|
|
|
|
(transduce xform + 0 (range 10))
|
|
;; 90
|
|
|
|
(eduction xform (range 10))
|
|
;; (0 2 4 6 8 10 12 14 16 18) ; an eduction
|
|
</code></pre></section><section><div data-markdown><script type="text/template">#### example 1... part 2</script></div><pre><code class="clojure">(def xform (map #(* 2 %)))
|
|
|
|
(into [1 2] xform (range 10))
|
|
;; [1 2 0 2 4 6 8 10 12 14 16 18]
|
|
|
|
(into (list 1 2) xform (range 10))
|
|
;; (18 16 14 12 10 8 6 4 2 0 1 2)
|
|
|
|
(sequence xform (range 10))
|
|
;; (0 2 4 6 8 10 12 14 16 18) ; lazy sec</code></pre></section><section><div data-markdown><script type="text/template">#### example 2</script></div><pre><code class="clojure">(def xform (comp
|
|
(map #(* 2 %))
|
|
(map #(+ 3 %))))
|
|
|
|
(sequence xform (range 10))
|
|
;; (3 5 7 9 11 13 15 17 19 21)</code></pre></section><section><div data-markdown><script type="text/template">#### example 2... weirdness</script></div><pre><code class="clojure">(def xform (map (comp
|
|
#(* 2 %)
|
|
#(+ 3 %))))
|
|
|
|
(sequence xform (range 10))
|
|
;; (6 8 10 12 14 16 18 20 22 24)</code></pre></section></section><section><section><h2>let's look at how we would implement the <em>map</em> function</h2></section><section><div data-markdown><script type="text/template">### Naive implementation of map</script></div><pre><code class="python">def naive_map(fn, lst):
|
|
ret = [0] * len(lst)
|
|
for i in range(len(lst)):
|
|
ret[i] = fn(lst[i])
|
|
return ret</code></pre><div data-markdown><script type="text/template">> I have no idea how to convert that to clojure</script></div></section><section><div data-markdown><script type="text/template">### clojuresque way of implementing map</script></div><pre><code class="clojure">(defn my-map [f coll]
|
|
(loop [ret [] coll coll]
|
|
(if (empty? coll)
|
|
ret
|
|
(recur (conj ret (f (first coll))) (rest coll)))))</code></pre></section><section><div data-markdown><script type="text/template">### `conj` is a weird beast</script></div><pre><code class="clojure">(conj [1 2 3] 99)
|
|
;; [1 2 3 99]</code></pre><pre><code class="clojure">(conj '(1 2 3) 99)
|
|
;; (99 1 2 3)</code></pre><pre><code class="clojure">(conj #{1 2 3} 99)
|
|
;; #{1 2 99 3}</code></pre><pre><code class="clojure">(conj {:key1 1 :key2 2} [:key3 3])
|
|
;; {:key1 1, :key2 2, :key3 3}</code></pre></section><section><div data-markdown><script type="text/template">### A more elegant way</script></div><pre><code class="clojure">(defn my-map [f coll]
|
|
(reduce #(conj %1 (f %2)) [] coll))</code></pre></section><section><div data-markdown><script type="text/template">#### many other map-like functions can be written similarly</script></div><pre><code class="clojure">(defn my-map [f coll]
|
|
(reduce #(conj %1 (f %2)) [] coll))</code></pre><pre><code class="clojure">(defn my-filter [pred coll]
|
|
(reduce #(if (pred %2) (conj %1 %2) %1) [] coll))</code></pre><pre><code class="clojure">(defn my-mapcat [pred coll]
|
|
(reduce (fn [r x](reduce conj r (f x))) [] coll))</code></pre></section><section><div data-markdown><script type="text/template">#### The main issue here the `conj`
|
|
|
|
* `conj` here yells "collection"
|
|
* `conj` creates intermediate data structures everywhere
|
|
</script></div></section></section><section><section><div data-markdown><script type="text/template">#### transducer functions!!</script></div><img src="/images/devil.png"></section><section><div data-markdown><script type="text/template">#### We start from where we were...</script></div><pre><code class="clojure">(defn my-map [f coll]
|
|
(reduce #(conj %1 (f %2)) [] coll))</code></pre><pre><code class="clojure">(defn my-filter [pred coll]
|
|
(reduce #(if (pred %2) (conj %1 %2) %1) [] coll))</code></pre><pre><code class="clojure">(defn my-mapcat [pred coll]
|
|
(reduce (fn [r x](reduce conj r (f x))) [] coll))</code></pre></section><section><div data-markdown><script type="text/template">#### And we extract the essence of our functions</script></div><pre><code class="clojure">(defn mapping [f]
|
|
(fn [step]
|
|
(fn [res x] (step res (f x)))))</code></pre><pre><code class="clojure">(defn filtering [pred]
|
|
(fn [step]
|
|
(fn [res x] (if (pred x) (step res x) res))))</code></pre><pre><code class="clojure">(def cat
|
|
(fn [step]
|
|
(fn [res x] (reduce step res x))))</code></pre><pre><code class="clojure">(defn mapcatting [f]
|
|
(comp cat (mapping f)))</code></pre></section><section><div data-markdown><script type="text/template">#### And we can now write our functions in the same way</script></div><pre><code class="clojure">(defn my-map [f coll]
|
|
(reduce ((mapping f) conj) [] coll))</code></pre><pre><code class="clojure">(defn my-filter [pred coll]
|
|
(reduce ((filtering pred) conj) [] coll))</code></pre><pre><code class="clojure">(defn my-mapcat [f coll]
|
|
(reduce ((mapcatting f) conj) [] coll))</code></pre></section><section><div data-markdown><script type="text/template">#### Note on mapcat</script></div><div data-markdown><script type="text/template">we have defined...</script></div><pre><code class="clojure">(defn mapping [f]
|
|
(fn [step]
|
|
(fn [res x] (step res (f x)))))</code></pre><pre><code class="clojure">(def cat
|
|
(fn [step]
|
|
(fn [res x] (reduce step res x))))</code></pre><pre><code class="clojure">(defn mapcatting [f]
|
|
(comp cat (mapping f)))</code></pre><p>for محمد بن موسی خوارزمی sake why is cat applied before map</p></section><section><div data-markdown><script type="text/template">#### develop mapcattig the reason is trivial</script></div><pre><code class="clojure">(fn [res x] (reduce mapping-step res x))</code></pre><pre><code class="clojure">(fn [res x] (reduce (fn [res x] (step res (f x))) res x))</code></pre></section><section><div data-markdown><script type="text/template">#### but for humans...</script></div><pre><code class="clojure">(def trans-1
|
|
(fn [step](fn [res x] (step res (+ 2 x)))))</code></pre><pre><code class="clojure">(def trans-2
|
|
(fn [step](fn [res x] (step res (* 3 x)))))</code></pre><pre><code class="clojure">(def trans-3
|
|
(comp trans-1 trans-2))</code></pre><pre><code class="clojure">(fn [res x] (fn [res x] (step res (* 3 x))) res (+ 2 x)))</code></pre></section><section data-markdown><script type="text/template">#### So transducers are...
|
|
* know nothing about the process they modify
|
|
* reducing functionnality fully encapsulates
|
|
* may call step 0, 1 or many times
|
|
* must pass previous result as next r otherwise must know nothing of r
|
|
* can transform input args</script></section></section><section><section data-markdown><script type="text/template">#### Stopping transducing process</script></section><section data-markdown><script type="text/template">#### goal
|
|
* transducers must be capable of calling 'completion'
|
|
* step functions must be capable of calling 'completion'</script></section><section data-markdown><script type="text/template">#### `reduce` already support stopping the process
|
|
* `(reduced 3)` wraps 3 in a Reduced wrapper
|
|
* `(reduced? x)` tells is a value is wrapped
|
|
* `(deref x)` or `@x` will get you the wrapped value</script></section><section data-markdown><script type="text/template">#### reducing process **must** the rules too
|
|
* if a step function returns a reduced value it
|
|
the processs must not supply anymore input
|
|
to the step function
|
|
* the dereference value is the final accumulated value</script></section><section><div data-markdown><script type="text/template">#### take-while example</script></div><pre><code class="clojure">(defn take-while [pred]
|
|
(fn [step]
|
|
(fn [res x]
|
|
(if (pred x)
|
|
(step res x)
|
|
(reduced res)))))</code></pre></section></section><section><section data-markdown><script type="text/template">#### Creating transducers that has state</script></section><section><div data-markdown><script type="text/template">#### thousand words example</script></div><pre><code class="clojure">(defn take [n]
|
|
(fn [step]
|
|
(let [nv (volatile! n)]
|
|
(fn [result input]
|
|
(let [n @nv
|
|
nn (vswap! nv dec)
|
|
result (if (pos? n) (step result input) result)]
|
|
(if (not (pos? nn))
|
|
(reduced result)
|
|
result))))))</code></pre></section></section><section><section data-markdown><script type="text/template">#### The transducer function **must** support two more arities</script></section><section data-markdown><script type="text/template">#### completion of process
|
|
When a process terminates, the arity one transducer function
|
|
is called to allow transducer to do extra things</script></section><section data-markdown><script type="text/template">#### initial reducing value
|
|
To allow initial values, trasducer **must** have an arity
|
|
zero function that calls the step function without any argument</script></section><section><div data-markdown><script type="text/template">real world example</script></div><pre><code class="clojure">(defn map [f]
|
|
(fn [rf]
|
|
(fn
|
|
([] (rf))
|
|
([result] (rf result))
|
|
([result input]
|
|
(rf result (f input))))))</code></pre></section></section><section><section data-markdown><script type="text/template">#### Completion can do thing at the very end</script></section><section data-markdown><script type="text/template">#### Arity one moment
|
|
* The completion of the process is done at arity one
|
|
* It **must** finish by calling the step function with the result (arity one)</script></section><section data-markdown><script type="text/template">#### Going to sleep...
|
|
https://github.com/clojure/clojure/blob/clojure-1.9.0-alpha14/src/clj/clojure/core.clj#L7115</script></section></section><section data-markdown><script type="text/template">questions?</script></section></div></div><script src="/reveal/js/reveal.js" type="text/javascript"></script><script src="/reveal/plugin/markdown/marked.js" type="text/javascript"></script><script src="/reveal/plugin/markdown/markdown.js" type="text/javascript"></script><script src="/reveal/plugin/math/math.js" type="text/javascript"></script><script src="/reveal/plugin/highlight/highlight.js" type="text/javascript"></script><script>Reveal.initialize({ slideNumber: true, history: false, rollingLinks: true });</script><script>hljs.initHighlightingOnLoad();</script></body></html> |