Learn iterator design patterns to help you greatly improve python performance

Hello, everyone, our git project has been updated, so we will continue to write about 1 design pattern.

The design pattern introduced to you today is very simple, called iterator, which is the iterator pattern. Iterator is a very important content in Python language. With the help of iterator, we can easily realize many complex functions. In deep learning, data acquisition is often realized through iterators. Therefore, this part of the content is very important, and it is recommended that everyone 1 must master it.

Simple case

Before we begin to introduce design patterns, let's look at a simple requirement. Suppose we now need to get the first few days of the week based on the variables passed in, for example, if we pass in 3 and return [Mon, Tue, Wed], we pass in 5 and return [Mon, Tue, Wed, Thu, Fri]. This requirement should be understood by everyone, and it is very, very simple.

If it is implemented with one function, this is the case:

def return_days(n):
    week = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
    return week[:n]

You can realize it by looking at 3 lines of code. Of course, there is no problem in writing this way in this problem scenario. But if we change the topic slightly, week here is not a fixed data, but read from the upstream or a certain file. n here is also a large number. Let's rewrite this function as follows:

def get_data(n):
    data = []
    for i in range(n):
        data.append(get_from_upstream())
    return data

We assume that the function get_from_upstream implements the specific logic of obtaining data, so what is the problem with the above function?

Some students will say that this is no problem, because other languages do the same when realizing data acquisition. Indeed, languages like Java may do this. But just because other languages are right, it doesn't mean Python is right. Because we didn't maximize the capabilities of Python.

There are two problems here. The first problem is delay, because as mentioned earlier, n is a large number. We get data from upstream, whether it is read through network or file, which is essentially IO operation, and the delay of IO operation is very large. Then it may take us a long time to collect all the n data, resulting in a long wait downstream. The second problem is memory, because we store this n data and return it once. If n is very large, the overhead pressure on memory is also very great. If the machine memory is not enough, it is likely to cause crash.

Then how to solve it?

In fact, the solution is very simple. If you are familiar with iterators, you will find that iterators are aimed at these two problems. We can rewrite the above logic into an iterator implementation, which is the iterator pattern.

iterator mode

Strictly speaking, iterator mode is only an application of iterator. It combines iterator with anonymous function very skillfully, and there are not many doorways to say. We rewrite the code just now for 1 time, and the details are all in the code.

def get_data(n):
    for i in range(n):
  yield get_from_upstream()


data_10 = lambda: get_data(10)
data_100 = lambda: get_data(100)

# use
for d in data_10:
    print(d)

It's very simple, but maybe you have to ask, since we have written the iterator get_data, it is not good to use it directly when we use it, for d in get_data (10). Why use anonymous function package layer 1 in the middle?

The reason is also very simple. If this data is used by ourselves, of course, there is no need for Tundish Layer 1. But if we pass it on to the downstream, it definitely doesn't want to consider too many details in the upstream, so the simpler the better. Therefore, we directly throw a packaged iterator in the past and directly call downstream. Otherwise, the downstream needs to sense the parameters passed in by the function get_data, which is obviously unreasonable.

The above is to learn iterator design patterns, help you greatly improve the performance of python details, more information about python iterator design patterns please pay attention to other related articles on this site!