C PLINQ Memory List Query Optimization Process

  • 2021-09-11 21:07:16
  • OfStack

In products (based on ASP. NET MVC development), it is necessary to often carry out drop-down matching and result query on drug names and name pinyin codes. In order to speed up the query, I added it to memory from the beginning (about 65,000 pieces of data).

Entity classes are attached below.


public class drugInfo
{
    public int drug_nameid    { get; set; }
    public string drug_name    { get; set; }
    public string drug_search_code    { get; set; }
}

Practice 1:


Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
key = key.ToLower();
var resultList = cacheList.Where(m => m.drug_name.ToLower().Contains(key) || m.drug_search_code.ToLower().Contains(key)).ToList();
stopWatch.Stop();
double eMseconds = Math.Max(0, stopWatch.Elapsed.TotalSeconds);

Refresh the page several times, and get an average time of about 35MS.

Practice 2:

In order to reduce the operation of CPU, we optimized the lowercase operation in LINQ expression by doing some actions on the cache list and storing the name and search code in lowercase first.

The following is an improved entity class.


public class drugInfo
{
    public int drug_nameid    { get; set; }
    public string drug_name    { get; set; }
    public string drug_search_code    { get; set; }
    public string lower_drug_name    { get; set; }
    public string lower_drug_search_code    { get; set; }
}
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
key = key.ToLower();
var resultList = cacheList.Where(m => m.lower_drug_name.Contains(key) || m.lower_drug_search_code.Contains(key)).ToList();
stopWatch.Stop();
double eMseconds = Math.Max(0, stopWatch.Elapsed.TotalSeconds);
ViewBag.useTime = string.Format(" Time {0} Seconds \r\n", eMseconds);

Refresh the page several times, and get an average time of about 16MS.

Although this will result in 1 more redundant data in the memory list, the performance improvement will be doubled.

Practice 3:

Enable the parallel computing of PLINQ. Parallel computing is the characteristic of NET 4.0. It can make use of the multi-core processing ability of CPU to improve the computing efficiency, but it must be multiplied
Generics such as LIST enable parallel computing as simple as using AsParallel (), with the following improvements:


Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
key = key.ToLower();
var resultList = cacheList.AsParallel().Where(m => m.lower_drug_name.Contains(key) || m.lower_drug_search_code.Contains(key)).ToList();
stopWatch.Stop();
double eMseconds = Math.Max(0, stopWatch.Elapsed.TotalSeconds);
ViewBag.useTime = string.Format(" Time {0} Seconds \r\n", eMseconds);

Similarly, we refresh the page several times more, and the average time we get is about 10MS.

Of course, at this point, everyone thought that this optimization was over, at least that's what I thought at that time.
---------------------------------------------------------------------------------------------------
But in fact, there was a big trouble.

Because the product runs on the server IIS, when using the AsParallel parallelism feature (by default, how many threads are used to execute PLINQ is determined by TPL when the program is running. However, if you need to limit the number of threads executing PLINQ queries (usually because multiple users are using the system at the same time, and in order for the server to serve as many users as possible at the same time, you must limit the system resources occupied by a single user), we can use the ParallelEnumerable. WithDegreeOfParallelism () extension method to do this.) The client consumes too many system resources with one request, causing the application pool to fake animation. Unable to provide service.

I also tried to use WithDegreeOfParallelism to set a relatively small value, but when using LOADRUNNER to turn on 200 concurrency, it will also produce suspended animation, so I have to try the method in Step 4 below.

Practice 4:


Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
key = key.ToLower();
ConcurrentBag<drugInfo> resultList = new ConcurrentBag<drugInfo>();
Parallel.For(0, cacheList.Count, new ParallelOptions { MaxDegreeOfParallelism = 4 }, (i) =>
{
var item = cacheList[i];
if (item.lower_drug_name.Contains(key) || item.lower_drug_search_code.Contains(key))
{
resultList.Add(item);
}
});
stopWatch.Stop();
double eMseconds = Math.Max(0, stopWatch.Elapsed.TotalSeconds);
ViewBag.useTime = string.Format(" Time {0} Seconds \r\n", eMseconds);

The timing is no different from Step 3, but it solves the problem of application pool fake animation when concurrent. At this point, the problem that has plagued for two days has been solved perfectly. Although using Parallel. For will bring about the problem of out-of-order results, the number of results is running out, and it doesn't matter to sort again.

See below for specific reasons:

ParallelOptions. MaxDegreeOfParallelism specifies the maximum number of threads that a parallel loop can use. When TPL starts to schedule and execute a parallel loop, it usually uses threads in the thread pool. At the beginning, if the threads in the thread pool are busy, it can provide a smaller number of threads for the parallel loop (but this number is at least 1, otherwise the parallel task cannot be executed and must be blocked and waited). By the time the threads in the thread pool have done a little work, the number of threads assigned to this parallel loop can be increased to speed up the completion of the entire task, but not more than the number specified by ParallelOptions. MaxDegreeOfParallelism.

WithDegreeOfParallelism () of PLINQ is different, and it must clearly indicate how many threads need to be used to complete the work. When an PLINQ query executes, the specified number of threads are immediately assigned to execute the query.

The reason PLINQ does not allow dynamic changes in the number of threads is that many PLINQ queries are "cascaded" and multiple participating threads must be synchronized to ensure correct results. If the number of threads is uncertain, it is very difficult to achieve thread synchronization.

About C # PLINQ memory list query optimization process this site to give you so much, I hope to help you!


Related articles: