Paging of Large Resultsets in ASP.NET
2008-11-11 10:32
363 查看
The paging of a large database resultset in Web applications is a well known problem. In short, you don't want all the results from your query to be displayed on a single Web page, so some sort of paged display is more appropriate. While it was not an easy task in the old ASP, the
There are numerous articles and posts concerning this problem and several proposed solutions. My goal here is not to present you with an amazing solves-it-all procedure, but to optimize all the existing methods and provide you with a testing application so you can do evaluation on your own. Here is a good starting point article which describes many different approaches and provides some performance test results:
How do I page through a recordset?
I was not satisfied with the most of them. First, half of the methods use old ADO and are clearly written for the "old" ASP. The rest of the methods are SQL server stored procedures. Some of them yield poor response times as you can see from the author’s performance results at the bottom of the page, but several have caught my attention.
The second problem with allowing other sorting columns beside the PK column(s) is that if those columns are not indexed in some way, none of these methods will help. In all of them a paged source must be sorted first and the cost of using ordering by non-indexed column is immense for large tables. The response times are so high that all the procedures are practically unusable in this case (the response varies from couple of seconds to couple of minutes depending on the size of the tables and the starting record being fetched). The indexing of other columns brings more performance issues and may be undesirable, for example it might significantly slow you down in a situation where you have a lot of daily imports.
ASP.NET DataGrid Paging Part 2 - Custom Paging
The methods in both articles could be optimized with just the Primary Key data copied to the temp table and then doing the join with the main query. Therefore, the essence of this method would be the following
Collapse
Copy Code
The method can be optimized further by copying the rows to the temp table until the end paging row is reached (
Collapse
Copy Code
Full Code – Paging_Asc_Desc
Collapse
Copy Code
Full Code – Paging_RowCount
Server-Side Paging with SQL Server
Although you'll need to be subscribed, a .zip file with the
Collapse
Copy Code
Full Code – Paging_SubQuery
Collapse
Copy Code
Full Code – Paging_Cursor
Collapse
Copy Code
The paging stored procedure call that returns the second page looks like this
Collapse
Copy Code
Note that in the original query, aliases are used in the
The
I used the auto generated large table for my tests and inserted around 500 000 records in it. If you don’t have a large table to experiment on, you can download the script for a table design and stored procedure for data generation here. I didn't want an identity column for my PK, I used the
The idea behind performance testing was to call a specific stored procedure many times through a loop and then measure the average response time. Also, in order to remove caching deviations and to model the real situation more accurately – multiple calls to a stored proc with the same page fetched each time seemed inappropriate. Thus, a random sequence of the same stored procedure with a set of different page numbers was required. Of course, a set of different page numbers assumes fixed number of pages (10 – 20) where each page would be fetched many times, but in a random sequence.
It’s not hard to notice that response times depend on the distance of the fetched page from the beginning of the resultset. The further the starting record is, more records need to be skipped. This is the reason I didn’t include first 20 pages in my random sequence. Instead I used the set of 2N pages. A loop was set to a (number of different pages)*1000. So, every page was fetched around 1000 times (more or less because of a random distribution).
DataGridcontrol in the ASP.NET simplifies this to a few lines of code. So, the paging is easy in ASP.NET, but the default behavior of the
DataGridis that all resulting records from your query will be fetched from SQL server to the ASP.NET application. If your query returns a million records this will cause some serious performance issues (if you need convincing, try executing such a query in your web application and see the memory consumption of the aspnet_wp.exe in the task manager). That's why a custom paging solution is required where desired behavior is to fetch only the rows from the current page.
There are numerous articles and posts concerning this problem and several proposed solutions. My goal here is not to present you with an amazing solves-it-all procedure, but to optimize all the existing methods and provide you with a testing application so you can do evaluation on your own. Here is a good starting point article which describes many different approaches and provides some performance test results:
How do I page through a recordset?
I was not satisfied with the most of them. First, half of the methods use old ADO and are clearly written for the "old" ASP. The rest of the methods are SQL server stored procedures. Some of them yield poor response times as you can see from the author’s performance results at the bottom of the page, but several have caught my attention.
Generalization
The three methods I decided to closely look into are the ones the author callsTempTable,
DynamicSQLand
Rowcount. I'll refer to the second method as the
Asc-Descmethod in the rest of this text. I don't think DynamicSQL was a good name, because you can apply dynamic SQL logic to the other methods too. The general problem with all these stored procedures is that you have to assess which columns you'll allow sorting for and that won't probably be just the PK column(s). This leads to a new set of problems – for each query you want to display via paging you must have as many different paging queries as you have different sorting columns. This means that you will either have a different stored procedure (regardless of the paging method applied) for each sorting column or you'll try to generalize this to only one stored procedure with the help of dynamic SQL. This has a slight performance impact, but increases maintainability if you need to display many different queries using this approach. Thus, I’ll try to generalize all of the stored procedures in this text with dynamic SQL, but in some cases it will be possible to achieve only a certain level of generalization, so you’ll still have to write separate stored procedures for some complex queries.
The second problem with allowing other sorting columns beside the PK column(s) is that if those columns are not indexed in some way, none of these methods will help. In all of them a paged source must be sorted first and the cost of using ordering by non-indexed column is immense for large tables. The response times are so high that all the procedures are practically unusable in this case (the response varies from couple of seconds to couple of minutes depending on the size of the tables and the starting record being fetched). The indexing of other columns brings more performance issues and may be undesirable, for example it might significantly slow you down in a situation where you have a lot of daily imports.
TempTable
The first one I would comment on is theTempTablemethod. This is actually a widely proposed solution and I encountered it several times. Here is another article that describes it along with the explanation and a sample how to use custom paging with the
DataGrid:
ASP.NET DataGrid Paging Part 2 - Custom Paging
The methods in both articles could be optimized with just the Primary Key data copied to the temp table and then doing the join with the main query. Therefore, the essence of this method would be the following
Collapse
Copy Code
CREATE TABLE #Temp ( ID int IDENTITY PRIMARY KEY, PK /* here goes PK type */ ) INSERT INTO #Temp SELECT PK FROM Table ORDER BY SortColumn SELECT ... FROM Table JOIN #Temp temp ON Table.PK = temp.PK ORDER BY temp.ID WHERE ID > @StartRow AND ID < @EndRow
The method can be optimized further by copying the rows to the temp table until the end paging row is reached (
SELECT TOP EndRow...), but the point is that in the worst case – for a table with 1 million records you end up with 1 million records in a temp table as well. Considering all this and having looked upon the results in the article above, I decided to discard this method from my tests.
Asc-Desc
This method uses default ordering in a subquery and then applies the reverse ordering. The principle goes like thisCollapse
Copy Code
DECLARE @temp TABLE ( PK /* PK Type */ NOT NULL PRIMARY ) INSERT INTO @temp SELECT TOP @PageSize PK FROM ( SELECT TOP (@StartRow + @PageSize) PK, SortColumn /*If sorting column is defferent from the PK, SortColumn must be fetched as well, otherwise just the PK is necessary */ ORDER BY SortColumn /* default order – typically ASC */) ORDER BY SortColumn /* reversed default order – typically DESC */ SELECT ... FROM Table JOIN @Temp temp ON Table.PK = temp.PK ORDER BY SortColumn /* default order */
Full Code – Paging_Asc_Desc
RowCount
The base logic of this method relies on the SQLSET ROWCOUNTexpression to both skip the unwanted rows and fetch the desired ones:
Collapse
Copy Code
DECLARE @Sort /* the type of the sorting column */ SET ROWCOUNT @StartRow SELECT @Sort = SortColumn FROM Table ORDER BY SortColumn SET ROWCOUNT @PageSize SELECT ... FROM Table WHERE SortColumn >= @Sort ORDER BY SortColumn
Full Code – Paging_RowCount
SubQuery
There are 2 more methods I’ve taken into consideration, and they come from different resources. The first one is well known triple query or theSubQuerymethod. The most thorough approach is the one I’ve found in the following article
Server-Side Paging with SQL Server
Although you'll need to be subscribed, a .zip file with the
SubQuerystored procedure variations is available. The Listing_04.SELECT_WITH_PAGINGStoredProcedure.txt file contains the complete generalized dynamic SQL. I used a similar generalization logic with all other stored procedures in this text. Here is the principle followed by the link to the whole procedure (I shortened the original code a bit, because a recordcount portion was unnecessary for my testing purposes).
Collapse
Copy Code
SELECT ... FROM Table WHERE PK IN (SELECT TOP @PageSize PK FROM Table WHERE PK NOT IN (SELECT TOP @StartRow PK FROM Table ORDER BY SortColumn) ORDER BY SortColumn) ORDER BY SortColumn
Full Code – Paging_SubQuery
Cursor
I’ve found the last method while browsing through the Google groups, you can find the original thread here. This method uses a server-side dynamic cursor. A lot of people tend to avoid cursors, they usually have poor performance because of their non-relational, sequential nature. The thing is that paging IS a sequential task and whatever method you use you have to somehow reach the starting row. In all the previous methods this is done by selecting all rows preceding the starting row plus the desired rows and then discarding all the preceding rows. Dynamic cursor has theFETCH RELATIVEoption which does the “magic” jump. The base logic goes like this
Collapse
Copy Code
DECLARE @PK /* PK Type */ DECLARE @tblPK TABLE ( PK /* PK Type */ NOT NULL PRIMARY KEY ) DECLARE PagingCursor CURSOR DYNAMIC READ_ONLY FOR SELECT @PK FROM Table ORDER BY SortColumn OPEN PagingCursor FETCH RELATIVE @StartRow FROM PagingCursor INTO @PK WHILE @PageSize > 0 AND @@FETCH_STATUS = 0 BEGIN INSERT @tblPK(PK) VALUES(@PK) FETCH NEXT FROM PagingCursor INTO @PK SET @PageSize = @PageSize - 1 END CLOSE PagingCursor DEALLOCATE PagingCursor SELECT ... FROM Table JOIN @tblPK temp ON Table.PK = temp.PK ORDER BY SortColumn
Full Code – Paging_Cursor
Generalization of Complex Queries
As pointed out before, all the procedures are generalized with dynamic SQL, thus, in theory, they can work with any kind of complex query. Here is a complex query sample that works withNorthwinddatabase.
Collapse
Copy Code
SELECT Customers.ContactName AS Customer, Customers.Address + ', ' + Customers.City + ', ' + Customers.Country AS Address, SUM([Order Details].UnitPrice*[Order Details].Quantity) AS [Total money spent] FROM Customers INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID INNER JOIN [Order Details] ON Orders.OrderID = [Order Details].OrderID WHERE Customers.Country <> 'USA' AND Customers.Country <> 'Mexico' GROUP BY Customers.ContactName, Customers.Address, Customers.City, Customers.Country HAVING (SUM([Order Details].UnitPrice*[Order Details].Quantity))>1000 ORDER BY Customer DESC, Address DESC
The paging stored procedure call that returns the second page looks like this
Collapse
Copy Code
EXEC ProcedureName /* Tables */ 'Customers INNER JOIN Orders ON Customers.CustomerID = Orders.CustomerID INNER JOIN [Order Details] ON Orders.OrderID = [Order Details].OrderID', /* PK */ 'Customers.CustomerID', /* ORDER BY */ 'Customers.ContactName DESC, Customers.Address DESC', /* PageNumber */ 2, /* Page Size */ 10, /* Fields */ 'Customers.ContactName AS Customer, Customers.Address + '', '' + Customers.City + '', '' + Customers.Country AS Address, SUM([Order Details].UnitPrice*[Order Details].Quantity) AS [Total money spent]', /* Filter */ 'Customers.Country <> ''USA'' AND Customers.Country <> ''Mexico''', /*Group By*/ 'Customers.CustomerID, Customers.ContactName, Customers.Address, Customers.City, Customers.Country HAVING (SUM([Order Details].UnitPrice*[Order Details].Quantity))>1000'
Note that in the original query, aliases are used in the
ORDER BYclause. You can't do that in paging procedures, because the most time-consuming task in all of them is skipping rows preceding the starting row. This is done in various ways, but the principle is not to fetch all the required fields at first, but only the PK column(s) (in case of
RowCountmethod the sorting column), which speeds up this task. All required fields are fetched only for the rows that belong to the requested page. Therefore, field aliases don't exist until the final query, and sorting columns have to be used earlier (in row skipping queries).
The
RowCountprocedure has another problem, it is generalized to work with only one column in the
ORDER BYclause. The same goes for
Asc-Descand
Cursormethods, though they can work with several ordering columns, but require that only one column is included in the PK. I guess this could be solved with more dynamic SQL, but in my opinion it is not worth the fuss. Although these situations are highly possible, they are not that frequent. Even if they are, you can always write a separate paging procedure following the principles above.
Performance Testing
I used these 4 methods in my tests, if you have a better one, I’d be glad to know about it. Nevertheless, I wanted to compare these methods and measure their performance. The first thought was to write an ASP.NET test application with paged DataGrid and then measure page response. Still, this wouldn’t reflect the true response time of the stored procedures, so the console application seemed more appropriate. I also included a web application, not for performance testing, but rather as an example of how DataGrid custom paging works with these stored procedures. They are both incorporated in the PagingTest Solution.I used the auto generated large table for my tests and inserted around 500 000 records in it. If you don’t have a large table to experiment on, you can download the script for a table design and stored procedure for data generation here. I didn't want an identity column for my PK, I used the
uniqueidentifierinstead. If you'll use this script, you may consider to add an identity after you generate the table. It will add numbers sorted by PK and you'll have an indication that correct page is fetched when you call a paging procedure with PK sorting.
The idea behind performance testing was to call a specific stored procedure many times through a loop and then measure the average response time. Also, in order to remove caching deviations and to model the real situation more accurately – multiple calls to a stored proc with the same page fetched each time seemed inappropriate. Thus, a random sequence of the same stored procedure with a set of different page numbers was required. Of course, a set of different page numbers assumes fixed number of pages (10 – 20) where each page would be fetched many times, but in a random sequence.
It’s not hard to notice that response times depend on the distance of the fetched page from the beginning of the resultset. The further the starting record is, more records need to be skipped. This is the reason I didn’t include first 20 pages in my random sequence. Instead I used the set of 2N pages. A loop was set to a (number of different pages)*1000. So, every page was fetched around 1000 times (more or less because of a random distribution).
相关文章推荐
- Paging of Large Resultsets in ASP.NET
- Paging of Large Resultsets in ASP.NET中介绍的SET ROWCOUNT方式存储过程的问题
- [转]Sorting, Filtering, and Paging with the Entity Framework in an ASP.NET MVC Application (3 of 10)
- It is not possible to run two different versions of ASP.NET in the same IIS process
- asp.net分页,Effective paging with List View control in ASP.NET
- ASP.NET中aspx页面runat="server"的本质(Essensial of runat=”server” in ASP.NET)
- Code snippet of the day : Resizing ASP.NET Datagrid Controls in Edit mode at runtime
- Front-line of Front-end, Rational Front-end Performance Tuning in ASP.NET MVC, III
- Grid View Paging and Sorting in ASP.Net
- Upload Large Files in ASP.NET Using HttpModule
- An Overview Of The New Services, Controls, And Features In ASP.NET 2.0
- IHttpActionResult – new way of creating responses in ASP.NET Web API 2
- Implementing the Repository and Unit of Work Patterns in an ASP.NET MVC Application (9 of 10)
- FW: How to spawn a process that runs under the context of the impersonated user in Microsoft ASP.NET pages
- Paging Database Results in ASP.NET (Prt2)(转载:http:
- Csharp:Paging Sorting Searching In ASP.NET MVC 5
- Can we call the Method of a controller from another controller in asp.net MVC?
- Front-line of Front-end, Rational Front-end Performance Tuning in ASP.NET MVC, I
- ASP.NET 2.0: Add build-in paging feature to repeater/为repeater添加内置分页功能
- Dynamic Generation of Word Document Report in ASP.NET with HTML