Generators

To infinity and beyond!

Vital Statistics

Name
Dave Jones
Twitter
@waveform80
GitHub
github.com/waveform80

So what is it?

  • An iterable defined by a function
  • A function which maintains state while it returns ("yields") multiple values
  • A sort of co-routine
  • All of the above!

So what is it?

  • You know how a function returns a value?
  • And you know how that value can be a sequence of things?
  • What if your function could return a sequence one piece at a time?
  • Without calling it multiple times …

So what is it?

Something like this...


            def alphabet():
                for i in range(26):
                    yield chr(ord('a') + i)
            

            >>> s = ""
            >>> for c in alphabet():
            ...     s += c
            ...
            >>> print(s)
            'abcdefghijklmnopqrstuvwxyz'
            >>> ''.join(alphabet())
            'abcdefghijklmnopqrstuvwxyz'
            

So what is it?

I've never seen one before …
no one has.
—Kryten

To infinity…

Of course, there's nothing stopping you from … erm … not stopping:


            def count(start=0):
                while True:
                    yield start
                    start += 1
            

            >>> for i in count():
            ...     print(i)
            ...
            

To infinity…

Here's another way of writing that example:


            from itertools import count
            

If it's good enough to be in the standard library, it's probably pretty useful …

Task 1: sample

Write a function to return a sample of an infinite iterator. It should have the following prototype:


            def sample(iterable, n=10):
                # Your code goes here
            

Remember: this isn't a generator function, just a regular one. We'll get onto the generators next …

Task 1: sample


            def sample(iterable, n=10):
                l = []
                for i in iterable:
                    if not n:
                        break
                    n -= 1
                    l.append(i)
                return l

            def sample(iterable, n=10):
                return [i for i, j in zip(iterable, range(n)]

            def sample(iterable, n=10):
                return [next(iterable) for i in range(n)]
            

Task 2: squares

Write a generator function to return the square numbers (1, 4, 9, 16, 25, …). It should have the prototype:


            def squares():
                # Your code goes here
            

Hint: You can use count() for this

Task 2: squares


            from itertools import count

            def squares():
                for i in count(1):
                    yield i * i
            

Task 3: triangles

Write a generator function to return the triangle numbers (1, 3, 6, 10, 15, …). It should have the prototype:


            def triangles():
                # Your code goes here
            

Hint: Use count() again

Task 4: fibonacci

Write a generator function to return the fibonacci sequence (1, 1, 2, 3, 5, 8, 13, …). It should have the prototype:


            def fib():
                # Your code goes here
            

Task 5: scaled

Write a generator function which, given an iterable, yields each element of iterable multiplied by n. It should have the prototype:


            def scaled(iterable, n):
                # Your code goes here
            

This function isn't terribly useful, but the concept is important for the next task...

Task 6: sliding_window

Write a generator function with the prototype:


            def sliding_window(iterable, n=2):
                # Your code goes here
            

This will yield a sliding window n elements long over iterable. For example, ABCDEF yields ABC, then BCD, then CDE, then DEF.

Task 7: differences

Write a generator function which returns the differences between values of an iterable:


            def differences(iterable):
                # Your code goes here
            

Hint: You can use sliding_window for this

Task 8: ratios

Write a generator function which returns the ratio between values of an iterable:


            def ratios(iterable):
                # Your code goes here
            

Hint: This should be really easy after differences

Task 9: Phi!

Find what the ratios of the Fibonacci sequence converge upon:


            >>> sample(ratios(fib()), 50)[-1]
            1.618033988749895
            

Extra credit: can you write a function that determines whether a sequence converges and if so, on what value?

Stock Market Analysis

The major challenge of the evening: use your new found generator skills to analyse (a rapid playback of the history of) the FTSE market. You'll find this at:

mrkrabs.waveform.org.uk/presentations/generators/stocks/market.html

Task 10: market_scraper

Build a generator function which, on each iteration, returns a new state of the market as a dict mapping stock symbols to their prices, changes, etc..


            def market_scraper(url="http://mrkrabs..."):
                while True:
                    # Some hints:
                    # use requests.get to grab the page
                    # use bs4.BeautifulSoup to extract data
                    # yield the data (if it's changed)
            

Task 11: portfolio_manager

Build a generator function which, given an iterable of market states (which you now have), yields buy and sell instructions.

Hints: use sliding_window to look at market states over time, and split out the buy/sell algorithms into their own generators.

Cool Examples


            import io
            import picamera
            from itertools import cycle

            def streams():
                # Infinitely cycle between two memory streams
                for stream in cycle((io.BytesIO(), io.BytesIO())):
                    stream.seek(0)
                    stream.truncate()
                    yield stream
                    # Do something with the stream here...

            with picamera.PiCamera() as camera:
                camera.capture_sequence(streams())
            

Cool Examples


            from picraft import Vector, Block

            def track_changes(frames, default=Block('air')):
                # yield dicts containing only those
                # vectors with changed blocks
                old_frame = None
                for frame in frames:
                    if old_frame is None:
                        old_frame = {v: default for v in frame}
                    changes = {
                        v: b for v, b in frame.items()
                        if old_frame.get(v) != b}
                    changes.update({
                        v: default for v in old_frame
                        if v not in frame})
                    yield changes
                    old_frame = frame
            

Cool Examples


            import io
            import sqlite3
            from lars import apache, sql

            connection = sqlite3.connect(
                'apache.db',
                detect_types=sqlite3.PARSE_DECLTYPES)

            with io.open('/var/log/apache2/access.log', 'rb') as infile:
                with io.open('apache.csv', 'wb') as outfile:
                    with apache.ApacheSource(infile) as source:
                        with sql.SQLTarget(
                                sqlite3, connection, 'log_entries',
                                create_table=True) as target:
                            for row in source:
                                target.write(row)
            

Vital Statistics

Name
Dave Jones
Occupation
Generator Gastronome
Twitter
@waveform80
GitHub
github.com/waveform80

Available Now!

from your nearest web-server

www.waveform.org.uk/presentations/generators/

Thank You

Questions? Pub!