Showing posts with label updating. Show all posts
Showing posts with label updating. Show all posts

Sunday, March 25, 2012

Basic advice on good practise required

I have master tables that I will be updating from our ERP system. Some examples I have seen take an approach of dropping a table in SQL server then creating it again before importing; some, and probably my choice, append and update; I have not seen an example where records are all deleted then the data appended afterwards. Of the three approaches which is generally regarded as best practise / most efficient?

The create and re-load versus the delete and then append are really the same thing from a process perspective, but they will have different performance due to the overhead of data change in SQL Server. Delete is a logged operation, so incurs overhead by writing the transaction log each row that is deleted. Dropping the table is just much faster. Truncating the table would be similar to the drop. It is an non-logged, non-selective delete in effect.

As to which is faster of the two methods, re-load or partial load and update, it depends on lots of factors. The speed of extract will be constant since both methods require you to extract all data, so it will come down to the cost of inserting lots of data, lots of write activity versus the cost of comparison in trying to reduce that write to only the changes required. I cannot give answers as things like row overall row width, comparison columns and how much data needs to be moved around to do each method will impact this. Here is an article that looks at two methods of comparing tables.

Get all from Table A that isn't in Table B
(http://www.sqlis.com/311.aspx)

Another option is to just insert all data, and capture failures due to a primary key violation. Then for the failures check or updates. I did some tests of a similar approach for patching bad data, and it was generally quicker to let it fail the insert and fix afterwards that check and potentially fix all rows. This will depend on the ratio of good to bad, or in your case, new rows, old rows, old that need to be updated.

If the cost of updates is high, then you will want to check if an update is really required, or has nothing changed. The Checksum Transform can be useful for this, as comparing an integer is faster than moving and comparing 200 columns of data to detect and change. Again it depends on your data.

Checksum Transformation
(http://www.sqlis.com/21.aspx)

The thing you need to work out is where is the bottle neck in your process. It may be that the extract query or network transfer is the slowest part and it does not matter which method you use for downstream loading.

My opinion is always go for deltas if you can, but if not then delete (drop/truncate) and reload all. It can be harder with foreign keys but it is going to be faster I think.

sql

Tuesday, March 20, 2012

Bad Update Performance with very large recordsets

When updating a table with 80 million records we are experiencing very bad
times. We found using maxdop1 to keep the process in one thread helpful,
but something is still causing problems.
Updating one bigint field which is nullable to a bigint value from a table
inner joined with the current table is crawling.
Should we remove the nullability of the field and set it to a default of -1?
Would this ensure space is allocated for the field? What else can we look
for?
What about NO LOCK or Serializable? Could
What would be the *fastest* way to update records in a table assuming only 1
connection to the database? How do fields allowing NULLs affect this?
Thanks.Never try to update 80 million rows at once if you can avoid it. Wrap the
updates in smaller batches of say 10, 50 or 100K in size. If no one else
needs to use the table while this is happening you can use TABLOCK hint to
help.
--
Andrew J. Kelly
SQL Server MVP
"Kory Yingling" <kory@.removeme-mlsc.com> wrote in message
news:u$j1RcBbDHA.1744@.TK2MSFTNGP12.phx.gbl...
> When updating a table with 80 million records we are experiencing very bad
> times. We found using maxdop1 to keep the process in one thread helpful,
> but something is still causing problems.
> Updating one bigint field which is nullable to a bigint value from a table
> inner joined with the current table is crawling.
> Should we remove the nullability of the field and set it to a default
of -1?
> Would this ensure space is allocated for the field? What else can we look
> for?
> What about NO LOCK or Serializable? Could
> What would be the *fastest* way to update records in a table assuming only
1
> connection to the database? How do fields allowing NULLs affect this?
> Thanks.
>

Saturday, February 25, 2012

Backups

Hi All,
We have setup Transactional Replication with queued updating subscribers option. Our Servers are in Failover cluster with windows 2000 AS. We are doing Full and Transactional log backups as usual. Can any one suggest what are the recovery scenarios, in ca
se of Publisher (if it is completely down, I mean all clusters are down) and User interactions to Subscriber immediately, so that data is more than of Publisher db.
Please provide us valuable suggestions.
Thanks,
John.
I'm not exactly sure what your question is. You loose me with this statement
:
"so that data is more than of Publisher db. "
With queued Updating, your Subscriber is essentially a mirror of your
Publisher's data. If your Publisher goes offline permanently you will lose
transactions that are on your publisher and haven't made their way back to
the Subscriber.
So, its not too hard to clone your Subscriber to rebuild your Publisher. To
do this you would have to drop your subscriber, copy the subscriber schema
and data to the publisher, re run your publication scripts on your new
publisher, and do a no-sync subscription.
If your Publisher goes offline for an extended period your chance of
bringing it back on line without conflicts increases. With Queued Updating
the conflict view only lets you view conflicts, not resolve them.
"John" <anonymous@.discussions.microsoft.com> wrote in message
news:1D2A989A-5347-4E74-AAD6-707E9EC9CE70@.microsoft.com...
> Hi All,
> We have setup Transactional Replication with queued updating subscribers
option. Our Servers are in Failover cluster with windows 2000 AS. We are
doing Full and Transactional log backups as usual. Can any one suggest what
are the recovery scenarios, in case of Publisher (if it is completely down,
I mean all clusters are down) and User interactions to Subscriber
immediately, so that data is more than of Publisher db.
> Please provide us valuable suggestions.
> Thanks,
> John.
>
|||The best suggestion I have is to look in BOL for the article about the
"synch with backup" option that can be used. This will not only walk you
through these scenarios, but explain how to create coherent, multi-server
backups.
Mike
Principal Mentor
Solid Quality Learning
"More than just Training"
SQL Server MVP
http://www.solidqualitylearning.com
http://www.mssqlserver.com