Basic advice on good practise required

I have master tables that I will be updating from our ERP system. Some examples I have seen take an approach of dropping a table in SQL server then creating it again before importing; some, and probably my choice, append and update; I have not seen an example where records are all deleted then the data appended afterwards. Of the three approaches which is generally regarded as best practise / most efficient?

The create and re-load versus the delete and then append are really the same thing from a process perspective, but they will have different performance due to the overhead of data change in SQL Server. Delete is a logged operation, so incurs overhead by writing the transaction log each row that is deleted. Dropping the table is just much faster. Truncating the table would be similar to the drop. It is an non-logged, non-selective delete in effect.

As to which is faster of the two methods, re-load or partial load and update, it depends on lots of factors. The speed of extract will be constant since both methods require you to extract all data, so it will come down to the cost of inserting lots of data, lots of write activity versus the cost of comparison in trying to reduce that write to only the changes required. I cannot give answers as things like row overall row width, comparison columns and how much data needs to be moved around to do each method will impact this. Here is an article that looks at two methods of comparing tables.

Get all from Table A that isn't in Table B
(http://www.sqlis.com/311.aspx)

Another option is to just insert all data, and capture failures due to a primary key violation. Then for the failures check or updates. I did some tests of a similar approach for patching bad data, and it was generally quicker to let it fail the insert and fix afterwards that check and potentially fix all rows. This will depend on the ratio of good to bad, or in your case, new rows, old rows, old that need to be updated.

If the cost of updates is high, then you will want to check if an update is really required, or has nothing changed. The Checksum Transform can be useful for this, as comparing an integer is faster than moving and comparing 200 columns of data to detect and change. Again it depends on your data.

Checksum Transformation
(http://www.sqlis.com/21.aspx)

The thing you need to work out is where is the bottle neck in your process. It may be that the extract query or network transfer is the slowest part and it does not matter which method you use for downstream loading.

My opinion is always go for deltas if you can, but if not then delete (drop/truncate) and reload all. It can be harder with foreign keys but it is going to be faster I think.

sql

Backup/copy to remote server

Hello,
I've spent the afternoon reading old newsgroup posts and BOL but am
still a bit confused about the best way to approach a new backup
scheme.
My organization would like to establish a single repository for backups
from our database servers, by either backing up databases on each
server and then moving them to the new, central server in some
automated fashion, or by simply pointing to the remote server as the
backup device.

>From a network performance standpoint I think backing up locally and
then copying the resulting files to a remote destination may be the
better option, as we have several servers and at least one of the
databases residing on them results in an almost 2GB backup file (with
most of the remainder smaller than that). I'm also unsure of exactly
how to go about this.
I believe SQL Server has been installed by default under the local
system account, so I'm wondering if this will limit my options with
regard to authentication across servers.
I've read about the BACKUP command, "log shipping" and other options,
but am not sure if any match our requirements.
If anyone could shine a light down the right path for me, my blood
pressure would be grateful. ;)vast55555@.yahoo.com wrote:
> Hello,
> I've spent the afternoon reading old newsgroup posts and BOL but am
> still a bit confused about the best way to approach a new backup
> scheme.
> My organization would like to establish a single repository for backups
> from our database servers, by either backing up databases on each
> server and then moving them to the new, central server in some
> automated fashion, or by simply pointing to the remote server as the
> backup device.
>
> then copying the resulting files to a remote destination may be the
> better option, as we have several servers and at least one of the
> databases residing on them results in an almost 2GB backup file (with
> most of the remainder smaller than that). I'm also unsure of exactly
> how to go about this.
> I believe SQL Server has been installed by default under the local
> system account, so I'm wondering if this will limit my options with
> regard to authentication across servers.
> I've read about the BACKUP command, "log shipping" and other options,
> but am not sure if any match our requirements.
> If anyone could shine a light down the right path for me, my blood
> pressure would be grateful. ;)
>
Create a share on your central server, configure SQL to run as a domain
user, grant that domain user permissions to the share, then use the
BACKUP command to backup directly to the UNC for that share. We move a
dozen backups around on our network each night, some of which are
approaching 200GB, there is no noticeable difference between backing up
locally and then copying vs. backing up over the network.
Tracy McKibben
MCDBA
http://www.realsqlguy.com|||Hi Tracy,
Thanks for your quick response. That sounds like the way to go. One
potential problem is that this is a hosted situation--not in-house--and
I think there may be no domain, but rather isolated machines (I'm not a
network engineer by trade so I'm a bit ignorant here).
Tracy McKibben wrote:
> Create a share on your central server, configure SQL to run as a domain
> user, grant that domain user permissions to the share, then use the
> BACKUP command to backup directly to the UNC for that share. We move a
> dozen backups around on our network each night, some of which are
> approaching 200GB, there is no noticeable difference between backing up
> locally and then copying vs. backing up over the network.
>
> --
> Tracy McKibben
> MCDBA
> http://www.realsqlguy.com|||Performance wise, backing up straight to a remote file share is often faster
than backing up locally and then copying the backup file to the same file
share. But there are other factors to consider. For instance, with backing u
p
straight to the remote file share, if it fails towards to the end due to som
e
network issue, the retry may take longer than just retrying the copy
operation.
Linchi
"vast55555@.yahoo.com" wrote:

> Hello,
> I've spent the afternoon reading old newsgroup posts and BOL but am
> still a bit confused about the best way to approach a new backup
> scheme.
> My organization would like to establish a single repository for backups
> from our database servers, by either backing up databases on each
> server and then moving them to the new, central server in some
> automated fashion, or by simply pointing to the remote server as the
> backup device.
>
> then copying the resulting files to a remote destination may be the
> better option, as we have several servers and at least one of the
> databases residing on them results in an almost 2GB backup file (with
> most of the remainder smaller than that). I'm also unsure of exactly
> how to go about this.
> I believe SQL Server has been installed by default under the local
> system account, so I'm wondering if this will limit my options with
> regard to authentication across servers.
> I've read about the BACKUP command, "log shipping" and other options,
> but am not sure if any match our requirements.
> If anyone could shine a light down the right path for me, my blood
> pressure would be grateful. ;)
>

Backup/copy to remote server

Hello,
I've spent the afternoon reading old newsgroup posts and BOL but am
still a bit confused about the best way to approach a new backup
scheme.
My organization would like to establish a single repository for backups
from our database servers, by either backing up databases on each
server and then moving them to the new, central server in some
automated fashion, or by simply pointing to the remote server as the
backup device.
>From a network performance standpoint I think backing up locally and
then copying the resulting files to a remote destination may be the
better option, as we have several servers and at least one of the
databases residing on them results in an almost 2GB backup file (with
most of the remainder smaller than that). I'm also unsure of exactly
how to go about this.
I believe SQL Server has been installed by default under the local
system account, so I'm wondering if this will limit my options with
regard to authentication across servers.
I've read about the BACKUP command, "log shipping" and other options,
but am not sure if any match our requirements.
If anyone could shine a light down the right path for me, my blood
pressure would be grateful. ;)vast55555@.yahoo.com wrote:
> Hello,
> I've spent the afternoon reading old newsgroup posts and BOL but am
> still a bit confused about the best way to approach a new backup
> scheme.
> My organization would like to establish a single repository for backups
> from our database servers, by either backing up databases on each
> server and then moving them to the new, central server in some
> automated fashion, or by simply pointing to the remote server as the
> backup device.
>>From a network performance standpoint I think backing up locally and
> then copying the resulting files to a remote destination may be the
> better option, as we have several servers and at least one of the
> databases residing on them results in an almost 2GB backup file (with
> most of the remainder smaller than that). I'm also unsure of exactly
> how to go about this.
> I believe SQL Server has been installed by default under the local
> system account, so I'm wondering if this will limit my options with
> regard to authentication across servers.
> I've read about the BACKUP command, "log shipping" and other options,
> but am not sure if any match our requirements.
> If anyone could shine a light down the right path for me, my blood
> pressure would be grateful. ;)
>
Create a share on your central server, configure SQL to run as a domain
user, grant that domain user permissions to the share, then use the
BACKUP command to backup directly to the UNC for that share. We move a
dozen backups around on our network each night, some of which are
approaching 200GB, there is no noticeable difference between backing up
locally and then copying vs. backing up over the network.
Tracy McKibben
MCDBA
http://www.realsqlguy.com|||Hi Tracy,
Thanks for your quick response. That sounds like the way to go. One
potential problem is that this is a hosted situation--not in-house--and
I think there may be no domain, but rather isolated machines (I'm not a
network engineer by trade so I'm a bit ignorant here).
Tracy McKibben wrote:
> Create a share on your central server, configure SQL to run as a domain
> user, grant that domain user permissions to the share, then use the
> BACKUP command to backup directly to the UNC for that share. We move a
> dozen backups around on our network each night, some of which are
> approaching 200GB, there is no noticeable difference between backing up
> locally and then copying vs. backing up over the network.
>
> --
> Tracy McKibben
> MCDBA
> http://www.realsqlguy.com|||Performance wise, backing up straight to a remote file share is often faster
than backing up locally and then copying the backup file to the same file
share. But there are other factors to consider. For instance, with backing up
straight to the remote file share, if it fails towards to the end due to some
network issue, the retry may take longer than just retrying the copy
operation.
Linchi
"vast55555@.yahoo.com" wrote:
> Hello,
> I've spent the afternoon reading old newsgroup posts and BOL but am
> still a bit confused about the best way to approach a new backup
> scheme.
> My organization would like to establish a single repository for backups
> from our database servers, by either backing up databases on each
> server and then moving them to the new, central server in some
> automated fashion, or by simply pointing to the remote server as the
> backup device.
> >From a network performance standpoint I think backing up locally and
> then copying the resulting files to a remote destination may be the
> better option, as we have several servers and at least one of the
> databases residing on them results in an almost 2GB backup file (with
> most of the remainder smaller than that). I'm also unsure of exactly
> how to go about this.
> I believe SQL Server has been installed by default under the local
> system account, so I'm wondering if this will limit my options with
> regard to authentication across servers.
> I've read about the BACKUP command, "log shipping" and other options,
> but am not sure if any match our requirements.
> If anyone could shine a light down the right path for me, my blood
> pressure would be grateful. ;)
>

Backup/copy to remote server

Hello,
I've spent the afternoon reading old newsgroup posts and BOL but am
still a bit confused about the best way to approach a new backup
scheme.
My organization would like to establish a single repository for backups
from our database servers, by either backing up databases on each
server and then moving them to the new, central server in some
automated fashion, or by simply pointing to the remote server as the
backup device.

>From a network performance standpoint I think backing up locally and
then copying the resulting files to a remote destination may be the
better option, as we have several servers and at least one of the
databases residing on them results in an almost 2GB backup file (with
most of the remainder smaller than that). I'm also unsure of exactly
how to go about this.
I believe SQL Server has been installed by default under the local
system account, so I'm wondering if this will limit my options with
regard to authentication across servers.
I've read about the BACKUP command, "log shipping" and other options,
but am not sure if any match our requirements.
If anyone could shine a light down the right path for me, my blood
pressure would be grateful. ;)
vast55555@.yahoo.com wrote:
> Hello,
> I've spent the afternoon reading old newsgroup posts and BOL but am
> still a bit confused about the best way to approach a new backup
> scheme.
> My organization would like to establish a single repository for backups
> from our database servers, by either backing up databases on each
> server and then moving them to the new, central server in some
> automated fashion, or by simply pointing to the remote server as the
> backup device.
> then copying the resulting files to a remote destination may be the
> better option, as we have several servers and at least one of the
> databases residing on them results in an almost 2GB backup file (with
> most of the remainder smaller than that). I'm also unsure of exactly
> how to go about this.
> I believe SQL Server has been installed by default under the local
> system account, so I'm wondering if this will limit my options with
> regard to authentication across servers.
> I've read about the BACKUP command, "log shipping" and other options,
> but am not sure if any match our requirements.
> If anyone could shine a light down the right path for me, my blood
> pressure would be grateful. ;)
>
Create a share on your central server, configure SQL to run as a domain
user, grant that domain user permissions to the share, then use the
BACKUP command to backup directly to the UNC for that share. We move a
dozen backups around on our network each night, some of which are
approaching 200GB, there is no noticeable difference between backing up
locally and then copying vs. backing up over the network.
Tracy McKibben
MCDBA
http://www.realsqlguy.com
|||Hi Tracy,
Thanks for your quick response. That sounds like the way to go. One
potential problem is that this is a hosted situation--not in-house--and
I think there may be no domain, but rather isolated machines (I'm not a
network engineer by trade so I'm a bit ignorant here).
Tracy McKibben wrote:
> Create a share on your central server, configure SQL to run as a domain
> user, grant that domain user permissions to the share, then use the
> BACKUP command to backup directly to the UNC for that share. We move a
> dozen backups around on our network each night, some of which are
> approaching 200GB, there is no noticeable difference between backing up
> locally and then copying vs. backing up over the network.
>
> --
> Tracy McKibben
> MCDBA
> http://www.realsqlguy.com
|||Performance wise, backing up straight to a remote file share is often faster
than backing up locally and then copying the backup file to the same file
share. But there are other factors to consider. For instance, with backing up
straight to the remote file share, if it fails towards to the end due to some
network issue, the retry may take longer than just retrying the copy
operation.
Linchi
"vast55555@.yahoo.com" wrote:

> Hello,
> I've spent the afternoon reading old newsgroup posts and BOL but am
> still a bit confused about the best way to approach a new backup
> scheme.
> My organization would like to establish a single repository for backups
> from our database servers, by either backing up databases on each
> server and then moving them to the new, central server in some
> automated fashion, or by simply pointing to the remote server as the
> backup device.
> then copying the resulting files to a remote destination may be the
> better option, as we have several servers and at least one of the
> databases residing on them results in an almost 2GB backup file (with
> most of the remainder smaller than that). I'm also unsure of exactly
> how to go about this.
> I believe SQL Server has been installed by default under the local
> system account, so I'm wondering if this will limit my options with
> regard to authentication across servers.
> I've read about the BACKUP command, "log shipping" and other options,
> but am not sure if any match our requirements.
> If anyone could shine a light down the right path for me, my blood
> pressure would be grateful. ;)
>