PHP array to millions of data to exclude duplicate data implementation code

If you get a list of uids with more than one million lines, the format is as follows:

 
10001000 
10001001 
10001002 
...... 
10001000 
...... 
10001111　　 

In fact, using the PHP array feature, it is very easy to arrange, let's take a look at the definition of PHP array: an array in PHP is actually an ordered map. A map is a type that associates values to keys. This type is optimized in many ways, so you can think of it as a real array, or list (vector), hash table (an implementation of mapping), dictionary, collection, stack, queue, and more. The value of an array element can also be another array. Tree structures and multidimensional arrays are also allowed.

In PHP arrays, keys, also known as indexes, are unique, and we can make use of this feature to rearrange. The sample code is as follows:

 
<?php 
//Defines an array to hold the reordered results
$result = array(); 
//Read the uid list file
$fp = fopen('test.txt', 'r'); 

while(!feof($fp)) 
{ 
$uid = fgets($fp); 
$uid = trim($uid); 
$uid = trim($uid, "r"); 
$uid = trim($uid, "n"); 

if($uid == '') 
{ 
continue; 
} 
//Use the uid as the key to see if the value exists
if(empty($result[$uid])) 
{ 
$result[$uid] = 1; 
} 
} 

fclose($fp); 

//Save the results to a file
$content = ''; 
foreach($result as $k => $v) 
{ 
$content .= $k."n"; 
} 
$fp = fopen('result.txt', 'w'); 
fwrite($fp, $content); 
fclose($fp); 
?>　　 

More than 20 lines of code, can carry on the arrangement heavy to the millions of above data, the efficiency is also good, very practical. Mobile phone number, email, also can use this way to arrange.

In addition, this method can also be used for two files to arrange the work, if you have two uid list files, the format is the same as the uid list above, the sample program is as follows:

 
<?php 
//Defines an array to hold the reordered results
$result = array(); 
//Read the first uid list file and put in $result_1
$fp = fopen('test_1.txt', 'r'); 
while(!feof($fp)) 
{ 
$uid = fgets($fp); 
$uid = trim($uid); 
$uid = trim($uid, "r"); 
$uid = trim($uid, "n"); 
if($uid == '') 
{ 
continue; 
} 
//Write $result with uid as key, and if there is a repeat, it will be overwritten
$result[$uid] = 1; 
} 
fclose($fp); 
//The second uid list file is read and rearranged
$fp = fopen('test_2.txt', 'r'); 
while(!feof($fp)) 
{ 
$uid = fgets($fp); 
$uid = trim($uid); 
$uid = trim($uid, "r"); 
$uid = trim($uid, "n"); 
if($uid == '') 
{ 
continue; 
} 
//Use the uid as the key to see if the value exists
if(empty($result[$uid])) 
{ 
$result[$uid] = 1; 
} 
} 
fclose($fp); 
//The result saved in $result will be reprinted and can be output to a file. The code is omitted
?> 

If you think about it, it's not hard to see that this feature of arrays can solve many more problems in our work.