PHP uses PDO to read a large amount of data from mysql

Preface

This article mainly introduces the related contents of PHP using PDO to read a large amount of data processing from mysql, and shares it for everyone's reference and study. The following words are not much to say, let's take a look at the detailed introduction.

Environment

mysql: 5.6.34 php: 5.6 nginx: php-fpm

Applicable scenario

Need to handle 1 fixed data set service

Service derivation of reading 1 fixed data from mysql Update and delete of mysql service operation that needs to be processed once More operations that need to deal with 1 set of data

pdo Key Settings

$dbh = new \PDO($dsn, $user, $pass);
#  Key settings, if not set, php Will still be from pdo1 Fetch data to php
$dbh->setAttribute(\PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
//perpare Cursor properties in are not required 
$sth = $dbh->prepare("SELECT * FROM `order`", array(\PDO::ATTR_CURSOR => \PDO::CURSOR_SCROLL));
$sth->execute();

Generator

Generator, iterative data operation

This generator can be omitted

Attempt code

class Test {
 public function test()
 {
  set_time_limit(0);
  $dbms='mysql';  // Database type 
  $host=C('DB_HOST'); // Database Hostname 
  $dbName=C('DB_NAME'); // Database used 
  $user=C('DB_USER');  // Database connection user name 
  $pass=C('DB_PWD');   // Corresponding password 
  $dsn="$dbms:host=$host;dbname=$dbName";
  $dbh = new \PDO($dsn, $user, $pass);
  $dbh->setAttribute(\PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
  $sth = $dbh->prepare("SELECT * FROM `order`");
  $sth->execute();
  $i = 0;

  $newLine = PHP_SAPI == 'cli' ? "\n" : '<br />';

  foreach ($this->cursor($sth) as $row) {
//   var_dump($row);
   echo $row['id'] . $newLine;
   $i++;
  }

  echo " Memory consumed: " . (memory_get_usage() / 1024 / 1024) . "M" . $newLine;
  echo " Number of rows of data processed: " . $i . $newLine;
  echo "success";
 }

 public function cursor($sth)
 {
  while($row = $sth->fetch(\PDO::FETCH_ASSOC)) {
   yield $row;
  }
 }
}

$test = new Test();
$test->test();

Output

1
... // Omitted part id
804288
 Memory consumed: "0.34918212890625M
 Number of rows of data processed: 254062
success

Summarize