Understand generator usage in python

  • 2020-06-19 10:44:52
  • OfStack

Generator (generator) concept

Instead of storing the results in a series, the generator stores the state of the generator, returning a value per iteration until it encounters an StopIteration exception.

Generator syntax

Generator expression: General list parsing syntax, but the list parsing [] replaced by ()
List parsing can do almost anything a generator expression can, except that when the sequence you need to process is large, list parsing is memory intensive.


>>> gen = (x**2 for x in range(5))
>>> gen
<generator object <genexpr> at 0x0000000002FB7B40>
>>> for g in gen:
...  print(g, end='-')
...
0-1-4-9-16-
>>> for x in [0,1,2,3,4,5]:
...  print(x, end='-')
...
0-1-2-3-4-5-

Generator functions: If the yield keyword appears in a function, the function is no longer a normal function, but a generator function.

But the generator function can produce a wireless sequence, so that the list cannot be processed at all.

What yield does is turn a function into an generator. A function with yield is no longer a normal function; the Python interpreter treats it as an generator.

The following is a generator function that produces odd Numbers indefinitely.


def
odd():
n=1
while
True:
yield
n
n+=2
odd_num
=
odd()
count
=
0
for
o
in
odd_num:
if
count
>=5:
break
print(o)
count
+=1

Of course, you can achieve a similar effect by writing an iterator manually, but the generator is more intuitive


class Iter:
  def __init__(self):
    self.start=-1
  def __iter__(self):
    return self
  def __next__(self):
    self.start +=2 
    return self.start
I = Iter()
for count in range(5):
  print(next(I))

As a side word: The generator contains the methods with EACH ___ 31en () and next__(), so you can iterate with for directly, and with self-made Iter without StopIteration you can only iterate manually


>>>
from
collections
import
Iterable
>>>
from
collections
import
Iterator
>>>
isinstance(odd_num,
Iterable)
True
>>>
isinstance(odd_num,
Iterator)
True
>>>
iter(odd_num)
is
odd_num
True
>>>
help(odd_num)
Help
on
generator
object:
odd
=
class
generator(object)
| Methods
defined
here:
|
| __iter__(self,
/)
|   Implement
iter(self).
|
| __next__(self,
/)
|   Implement
next(self).
......

With the results above, you can now loop with confidence in Iterator.

In for loop executes, each loop will execute fab functions within the code, and performs to yield b, fab function returns a value iteration, the next iteration, code from yield b under 1 statement to continue, and the function of the local variable seems to interrupt execution and the last is completely 1 sample before, so the function to continue, until meet yield again. It looks as if a function has been interrupted by yield several times during normal execution, with each interruption returning the current iteration value through yield.

yield and return

In 1 generator, if there is no return, the default is to return StopIteration at the end of the function.


>>> def g1():
...   yield 1
...
>>> g=g1()
>>> next(g)  # The first 1 Time to call next(g) At the end of the execution yield Statement is suspended, so the program is not finished executing. 
1
>>> next(g)  # The program attempts to yield Under the statement 1 A statement starts to execute, finds the end of the statement, and throws it StopIteration The exception. 
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration
>>>

If return is encountered and return is executed, StopIteration is thrown to terminate the iteration.


>>>
def
g2():
...  
yield
'a'
...  
return
...  
yield
'b'
...
>>>
g=g2()
>>>
next(g)  # The program stays at the end of execution yield
 'a' The position after the statement. 
'a'
>>>
next(g)  # Program discovery 1 Statement is return , so throw out StopIteration Exception, like this yield
 'b' The statement is never executed. 
Traceback
(most
recent
call
last):
 File
"<stdin>",
line
1,
in
<module>
StopIteration

If a value is returned after return, then the value is an exception to StopIteration, not the return value of the program.

The generator has no way to return a value using return.


>>> def g3():
...   yield 'hello'
...   return 'world'
...
>>> g=g3()
>>> next(g)
'hello'
>>> next(g)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration: world

Methods supported by the generator


>>>
help(odd_num)
Help
on
generator
object:
odd
=
class
generator(object)
| Methods
defined
here:
......
| close(...)
|   close()
->
raise
GeneratorExit
inside
generator.
|
| send(...)
|   send(arg)
->
send
'arg'
into
generator,
|   return
next
yielded
value
or
raise
StopIteration.
|
| throw(...)
|   throw(typ[,val[,tb]])
->
raise
exception
in
generator,
|   return
next
yielded
value
or
raise
StopIteration.
......

close()

Turn off the generator function manually, and subsequent calls return the StopIteration exception directly.


>>> def g4():
...   yield 1
...   yield 2
...   yield 3
...
>>> g=g4()
>>> next(g)
1
>>> g.close()
>>> next(g)  # Shut down, yield 2 and yield 3 The statement will no longer work 
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration

send()

The main feature of the generator function is that it can accept 1 variable passed in from the outside and return the result based on its contents.

This is the hardest part of the generator function to understand, and the most important part of implementing the coroutines that I'll talk about later.


def
gen():
  value=0
  while
True:
    receive=yield
value
    if
receive=='e':
      break
    value
=
'got: %s'
%
receive
g=gen()
print(g.send(None)) 
print(g.send('aaa'))
print(g.send(3))
print(g.send('e'))

Execution process:

The generator function can be started via g.send (None) or next(g) and executed to the end of the first yield statement. At this point, the yield statement is executed, but no receive is assigned. Note: only send(None) can be used when starting the generator function. Any attempt to enter another value will get an error message.

With ES113en.send (' aaa'), aaa is passed in and assigned to receive, then the value of value is calculated and returned to the head of while, executing yield value statement with stop. At this point, yield value prints "got: aaa" and hangs.

Step 2 is repeated through ES129en. send(3), and the final output is "got: 3".

When we have ES135en.send (' e'), the program executes break and then pushes out of the loop, and finally the whole function executes, so we get an StopIteration exception.

The final execution results are as follows:


def
odd():
n=1
while
True:
yield
n
n+=2
odd_num
=
odd()
count
=
0
for
o
in
odd_num:
if
count
>=5:
break
print(o)
count
+=1
0

throw()

Used to send an exception to the generator function that ends either a system-defined exception or a custom exception.

throw() runs out of the exception and ends the program, either by consuming 1 yield, or by ending the program without another yield.


def
odd():
n=1
while
True:
yield
n
n+=2
odd_num
=
odd()
count
=
0
for
o
in
odd_num:
if
count
>=5:
break
print(o)
count
+=1
1

The output result is:


def
odd():
n=1
while
True:
yield
n
n+=2
odd_num
=
odd()
count
=
0
for
o
in
odd_num:
if
count
>=5:
break
print(o)
count
+=1
2

Explanation:

print(next(g)) : output normal value and stay before yield 'normal value 2'.

Since g.throw(ValueError) is executed, all subsequent try statements are skipped, that is, yield 'normal value 2' is not executed, then enter the except statement and print out we got ValueError here. Then enter the while statement section again, consume 1 yield, so output normal value.

print(next(g)), executes the yield 'normal value 2' statement and stays where it is after executing the statement.

g.throw(TypeError) : jumps out of the try statement so that print(' here') is not executed, then executes the break statement, jumps out of the while loop, and then reaches the end of the program, so runs out of the StopIteration exception.

Here is a general example to expand or flatten a multidimensional list.


def
flatten(nested):
  try:
    # If it is a string, throw it manually TypeError . 
    if
isinstance(nested,
str):
      raise
TypeError
    for
sublist
in
nested:
      #yield
 flatten(sublist)
      for
element
in
flatten(sublist):
        #yield
 element
        print('got:',
element)
  except
TypeError:
    #print('here')
    yield
nested
L=['aaadf',[1,2,3],2,4,[5,[6,[8,[9]],'ddf'],7]]
for
num
in
flatten(L):
  print(num)

If it is a bit difficult to understand, then it is clear to leave the comments on the print statement open.

conclusion

According to duck model theory, a generator is an iterator that can be iterated using for.

On the first execution of next(generator), the program is suspended after the yield statement is executed, and all parameters and states are saved. When next(generator) is executed one more time, it is executed after the pending state. The loop ends when the end of the program or StopIteration is encountered.

Parameters can be passed in via ES235en.send (arg), which is the coroutine model.

You can pass in an exception via ES241en. throw(exception). The throw statement consumes 1 yield. The generator can be turned off manually by generator.close ().

next() is equivalent to send(None)


Related articles: