Use PyV8 to execute js code in the Python crawler

preface

Maybe a lot of people will think that this is a weird demand, why don't the crawler crawl the data well? Why do you parse js? Are you stuffed?

Search 1 on the Internet about this question is really a lot, but most of the children's shoes because their js foundation is too bad, or HTML foundation is bad, or ajax foundation is bad, anyway all aspects are bad. The foundation so slag not to learn the foundation to write what crawler?

Then you must ask, "my friend, why do you have this need for TM? Are you a piece of technical junk?"

No, no, no, the blogger, as a siege corpse with more than 3 years of front-end experience, how could he be stumped by this problem? The problem that the old husband encountered today is obviously not so simple.

The problem

So what's wrong with bloggers?

The blogger is going to climb an interface today, but calls that interface with a token, which is a token like thing stored in Cookie. The value of Cookie is generated by a segment of js, which is obtained through another interface, and the code of js is still dynamic, WTF!! Developers, what are you doing here?

Passerby a: I suck, claiming that experienced bloggers can't analyze the logic of js?

Yeah, I just don't. The js code is confusing and encrypted. You're blind.

Well, I'll just execute and get the results, whatever he wrote.

Train of thought

Reason 1 reason train of thought, the thing that should do now is very simple actually

Request the interface A and get the dynamically generated obfuscated js code Execute the js code and get the generated cookie value Request interface B with the token generated by js Get the results, have fun...

The train of thought is quite clear, the feeling can be realized in seconds. (a)

problem

Python inside js? Interesting. Why don't I use nodejs?

Because Python is the best language in the world! No one!

Found the magic module PyV8, the machine already has pip, do not perform the installation 1 to OK?

pip install pyv8

Do not doubt, the blogger machine is installed with Kali Linux, Root permission, do not need sudo

Then an error

pip install -U PyV8
Collecting PyV8
 Using cached PyV8-0.5.zip
Building wheels for collected packages: PyV8
 Running setup.py bdist_wheel for PyV8 ... error
 Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-QUm4bX/PyV8/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpb0udlepip-wheel- --python-tag cp27:
 running bdist_wheel
 running build
 running build_py
 creating build
 creating build/lib.linux-x86_64-2.7
 copying PyV8.py -> build/lib.linux-x86_64-2.7
 running build_ext
 building '_PyV8' extension
 creating build/temp.linux-x86_64-2.7
 creating build/temp.linux-x86_64-2.7/src
 x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-cFt4xx/python2.7-2.7.12=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DBOOST_PYTHON_STATIC_LIB -Ilib/python/inc -Ilib/boost/inc -Ilib/v8/inc -I/usr/include/python2.7 -c src/Exception.cpp -o build/temp.linux-x86_64-2.7/src/Exception.o
 cc1plus: warning: command line option  ' -Wstrict-prototypes' is valid for C/ObjC but not for C++
 In file included from src/Exception.cpp:1:0:
 src/Exception.h:6:16: fatal error: v8.h:  There is no file or directory 
 #include <v8.h>
     ^
 compilation terminated.
 error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
 
 ----------------------------------------
 Failed building wheel for PyV8
 Running setup.py clean for PyV8
Failed to build PyV8
Installing collected packages: PyV8
 Running setup.py install for PyV8 ... error
 Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-QUm4bX/PyV8/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-7OAwUa-record/install-record.txt --single-version-externally-managed --compile:
 running install
 running build
 running build_py
 creating build
 creating build/lib.linux-x86_64-2.7
 copying PyV8.py -> build/lib.linux-x86_64-2.7
 running build_ext
 building '_PyV8' extension
 creating build/temp.linux-x86_64-2.7
 creating build/temp.linux-x86_64-2.7/src
 x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-cFt4xx/python2.7-2.7.12=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DBOOST_PYTHON_STATIC_LIB -Ilib/python/inc -Ilib/boost/inc -Ilib/v8/inc -I/usr/include/python2.7 -c src/Exception.cpp -o build/temp.linux-x86_64-2.7/src/Exception.o
 cc1plus: warning: command line option  ' -Wstrict-prototypes' is valid for C/ObjC but not for C++
 In file included from src/Exception.cpp:1:0:
 src/Exception.h:6:16: fatal error: v8.h:  There is no file or directory 
  #include <v8.h>
     ^
 compilation terminated.
 error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
 
 ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-QUm4bX/PyV8/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-7OAwUa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-QUm4bX/PyV8/

It seems that the file v8.h is missing, but I don't know what it means.

To solve

The solution was found through the search engine. It turned out that PyV8 depended on Boost. However, the problem was not officially stated, so we had to install this package first

apt-get update && apt-get install libboost-all-dev

After the installation is completed, continue to install PyV8, the same problem as above, it seems that you can only come manually.

Download https: / / github com/emmetio/pyv8 - binaries

Unzip and select the appropriate file for your system environment. Unzip again and copy the extracted file to

/usr/lib/python2.7/dist-packages/

Go inside, then test to see if it succeeds, terminal execution

python
import PyV8

If there are no errors, then I've done it and have fun. Here's the js code I need to parse

var l = [119, 98, 115, 33, 111, 109, 120, 105, 118, 62, 92, 50, 50, 54, 45, 50, 50, 51, 45, 50, 50, 55, 45, 50, 49, 58, 45, 50, 50, 49, 45, 50, 51, 51, 45, 50, 50, 52, 45, 50, 50, 51, 45, 50, 50, 54, 45, 50, 49, 55, 45, 50, 49, 58, 45, 50, 49, 50, 45, 50, 50, 54, 45, 50, 50, 58, 45, 50, 50, 49, 45, 50, 50, 51, 45, 50, 50, 58, 45, 50, 51, 51, 45, 50, 50, 58, 45, 50, 50, 55, 45, 50, 50, 54, 45, 50, 50, 54, 94, 60, 119, 98, 115, 33, 121, 119, 99, 100, 108, 62, 92, 49, 45, 51, 50, 45, 53, 45, 55, 45, 50, 50, 45, 57, 45, 56, 45, 50, 51, 45, 51, 45, 51, 49, 45, 50, 52, 45, 50, 54, 45, 50, 49, 45, 50, 57, 45, 52, 45, 58, 45, 50, 53, 45, 50, 56, 45, 54, 45, 50, 55, 45, 50, 58, 45, 50, 94, 60, 119, 98, 115, 33, 118, 62, 35, 35, 60, 103, 112, 115, 33, 41, 119, 62, 49, 60, 119, 61, 121, 119, 99, 100, 108, 47, 109, 102, 111, 104, 117, 105, 60, 119, 44, 44, 42, 124, 118, 44, 62, 84, 117, 115, 106, 111, 104, 47, 103, 115, 112, 110, 68, 105, 98, 115, 68, 112, 101, 102, 41, 111, 109, 120, 105, 118, 92, 121, 119, 99, 100, 108, 92, 119, 94, 94, 42, 126, 60, 37, 47, 100, 112, 112, 108, 106, 102, 41, 40, 114, 117, 112, 108, 102, 111, 40, 45, 118, 45, 124, 113, 98, 117, 105, 59, 40, 48, 40, 126, 42, 60];
eval(function(p, a, c, k, e, d) {
 e = function(c) {
 return (c < a ? "" : e(parseInt(c / a))) + ((c = c % a) > 35 ? String.fromCharCode(c + 29) : c.toString(36))
 };
 if (!''.replace(/^/, String)) {
 while (c--) d[e(c)] = k[c] || e(c);
 k = [function(e) {
 return d[e]
 }];
 e = function() {
 return '\\w+'
 };
 c = 1
 };
 while (c--) if (k[c]) p = p.replace(new RegExp('\\b' + e(c) + '\\b', 'g'), k[c]);
 return p
}('6 3=\'\';7(2=0;2<4.5;2++){3+=8.a(4[2]-1)};9(3)', 11, 11, '||i|t|l|length|var|for|String|eval|fromCharCode'.split('|'), 0, {}))

It's been sorted out, but it's only one line at the beginning, which is awkward

posture

There have been many twists and turns along the way, but there have been a number of postures. For example, how to restore the confused js to its original code

You can easily solve this problem by using the Firebug plug-in. Open the firebug plug-in, find the script option, and select the item with eval

var balwi=[115,116,115,122,112,115,110,106,122,110,122,112,101,119,115,106,113,101,116,116,119,106];var ljpry=[15,21,4,9,12,14,11,0,18,20,8,16,7,2,1,10,17,13,19,6,5,3];var j="";for (k=0;k<ljpry.length;k++){j+=String.fromCharCode(balwi[ljpry[k]])};$.cookie('qtoken',j,{path:'/'});

After a little bit of tidying up 1, you get a well-formed code

var balwi = [115, 116, 115, 122, 112, 115, 110, 106, 122, 110, 122, 112, 101, 119, 115, 106, 113, 101, 116, 116, 119, 106];
var ljpry = [15, 21, 4, 9, 12, 14, 11, 0, 18, 20, 8, 16, 7, 2, 1, 10, 17, 13, 19, 6, 5, 3];
var j = "";
for (k = 0; k < ljpry.length; k++) {
 j += String.fromCharCode(balwi[ljpry[k]])
};
$.cookie('qtoken', j, {
 path: '/'
});

With the original code, it is easy to get the token generation algorithm, using Python generation, this time without the bother of PyV8.

conclusion

The above is the whole content of this article, I hope the content of this article to your study or work can bring 1 definite help, if you have questions you can leave a message to communicate.