How does python3 read the npy file of python2

  • 2021-10-15 10:59:33
  • OfStack

python3 will report an error when reading python2 packaged npy file, because the encoding method is different, so it is only necessary to add the encoding method when reading.

Solution


docs_train = np.load('./data/20news_clean/train.txt.npy', allow_pickle=True, encoding='bytes')
docs_test = np.load('./data/20news_clean/test.txt.npy', allow_pickle=True, encoding='bytes')

The files in the path are packaged by python2.

Addition: Error loading Python 2. npy file in Python 3

I have. npy files, which are np. save ('filename') created by command using Python 2.7. 9 and Numpy version 1.11. 3. These files are generated on an external computer that is part 1 of our institute's linux cluster.

I copy the files to the local computer so that I can import them np. load ('filename. npy'). On my local computer, I am running Python 3.5. 2 with Jupyter-Notebook and Numpy version 1.13. 0.

The native operating system is Ubuntu 16.04.2.

When I tried to load the file locally, an error occurred:


ValueError: invalid literal for int() with base 16

After browsing through some Stackoverflow problems, I tried to specify the encoding in the following way:


np.load('filename.npy',encoding='latin1')

This gives the same error. encoding = 'bytes' Yield:


TypeError: can't multiply sequence by non-int of type 'float'

This is the larger fragment of Traceback:


/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
417       else:
418         return format.read_array(fid, allow_pickle=allow_pickle,
--> 419                     pickle_kwargs=pickle_kwargs)
420     else:
421       # Try a pickle

/usr/local/lib/python3.5/dist-packages/numpy/lib/format.py in read_array(fp, allow_pickle, pickle_kwargs)
638       pickle_kwargs = {}
639     try:
--> 640       array = pickle.load(fp, **pickle_kwargs)
641     except UnicodeError as err:
642       if sys.version_info[0] >= 3:

/usr/local/lib/python3.5/dist-packages/sympy/core/numbers.py in __new__(cls, num, prec)
823         else:
824           _mpf_ = mpmath.mpf(
--> 825             S.NegativeOne**num[0]*num[1]*2**num[2])._mpf_
826     elif isinstance(num, Float):
827       _mpf_ = num._mpf_

TypeError: can't multiply sequence by non-int of type 'float'

I guess there is a coding problem when converting between Python and Numpy versions. Any thoughts on how to import files?

Solutions

As shown in, *. What is the storage method of data in npy? The. npy file is ByteCode, which you will see if you open 1 ByteCode in the 106-ary editor.

Python 2 ByteCode. pyc,. pyo files cannot run in Python 3 because the virtual machine and compiler builds have changed with the major version.

Similarly, the C internal structure of NumPy and the ByteCode compiler have changed in Python 3, thus breaking backward compatibility. (This is intentional, because ByteCode is not meant to last that long, or it cannot be used with the version 1 created.)

The composition of these changes means that these Python 2. npy files cannot be used without major changes to the ByteCode interpreter of Python 3 and NumPy of Python 3, and/or the translator from Python 2 NumPy ByteCode to Python 3. Python 3.

As I mentioned earlier, this is a bit like the X/Y problem. You should not rely on. npy files to work between versions, because there is no guarantee that they will remain 1, because they are inherently volatile (for example, ES90VM bytecode).

Instead of reverse engineering ByteCode to debug it, try to get the source that generated these files.


Related articles: