Basic advice on good practise required

I have master tables that I will be updating from our ERP system. Some examples I have seen take an approach of dropping a table in SQL server then creating it again before importing; some, and probably my choice, append and update; I have not seen an example where records are all deleted then the data appended afterwards. Of the three approaches which is generally regarded as best practise / most efficient?

The create and re-load versus the delete and then append are really the same thing from a process perspective, but they will have different performance due to the overhead of data change in SQL Server. Delete is a logged operation, so incurs overhead by writing the transaction log each row that is deleted. Dropping the table is just much faster. Truncating the table would be similar to the drop. It is an non-logged, non-selective delete in effect.

As to which is faster of the two methods, re-load or partial load and update, it depends on lots of factors. The speed of extract will be constant since both methods require you to extract all data, so it will come down to the cost of inserting lots of data, lots of write activity versus the cost of comparison in trying to reduce that write to only the changes required. I cannot give answers as things like row overall row width, comparison columns and how much data needs to be moved around to do each method will impact this. Here is an article that looks at two methods of comparing tables.

Get all from Table A that isn't in Table B
(http://www.sqlis.com/311.aspx)

Another option is to just insert all data, and capture failures due to a primary key violation. Then for the failures check or updates. I did some tests of a similar approach for patching bad data, and it was generally quicker to let it fail the insert and fix afterwards that check and potentially fix all rows. This will depend on the ratio of good to bad, or in your case, new rows, old rows, old that need to be updated.

If the cost of updates is high, then you will want to check if an update is really required, or has nothing changed. The Checksum Transform can be useful for this, as comparing an integer is faster than moving and comparing 200 columns of data to detect and change. Again it depends on your data.

Checksum Transformation
(http://www.sqlis.com/21.aspx)

The thing you need to work out is where is the bottle neck in your process. It may be that the extract query or network transfer is the slowest part and it does not matter which method you use for downstream loading.

My opinion is always go for deltas if you can, but if not then delete (drop/truncate) and reload all. It can be harder with foreign keys but it is going to be faster I think.

sql

Bandwidth required for remote sites to use Sharepoint Portal se

If I have a sharepoint installation at my head office and remote
sites with users from about 20-80 need to access this Intranet.
How much bandwith is suggested to accomodate users of that size ? At
two sites I have a 768k frame relay and at the largest remote site with
about 80 users there is a 10MB wan connection to head office.
This question should be posted to the sharepoint newsgroup.
http://www.zetainteractive.com - Shift Happens!
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com
"scuba19" <scuba19@.discussions.microsoft.com> wrote in message
news:A335E9FE-5EF5-4416-8E89-6E6137A9F90A@.microsoft.com...
> If I have a sharepoint installation at my head office and remote
> sites with users from about 20-80 need to access this Intranet.
> How much bandwith is suggested to accomodate users of that size ?
> At
> two sites I have a 768k frame relay and at the largest remote site
> with
> about 80 users there is a 10MB wan connection to head office.
>

Bandwidth Required For ODBC Access

Hi,
I need to setup an ODBC link over our Internet connection between MS Access
(local) and a mysql server (Remote - the local PC has the mysql ODBC driver
installed). What kind of bandwidth requirements am I looking at? Will it
function over a 64k leased line? If this would be slow, could there be any
major impact on other uses such as web and email?
Regards,
DominicIn message <c08fh8$fmc$1$8300dec7@.news.demon.co.uk>, Dominic Martin
<dominic.martin@.dplm.co.uk> writes
>Hi,
>I need to setup an ODBC link over our Internet connection between MS Access
>(local) and a mysql server (Remote - the local PC has the mysql ODBC driver
>installed). What kind of bandwidth requirements am I looking at? Will it
>function over a 64k leased line? If this would be slow, could there be any
>major impact on other uses such as web and email?
It's certainly feasible to run an ODBC connection over a 64k link, and I
have done so. You haven't really given enough information to answer the
question. You need to know how many data requests will be issued and how
much data will have to be retrieved for each request.
You may need to consider the time distribution of the requests, will
they all occur at the same time? How long are the users willing to wait
for their data? What else will be using the line at the same time? Will
those users be prepared to wait for their data?
Bernard Peek
London, UK. DBA, Manager, Trainer & Author. Will work for money.|||"Bernard Peek" <bap@.shrdlu.com> wrote in message
> It's certainly feasible to run an ODBC connection over a 64k link, and I
> have done so. You haven't really given enough information to answer the
> question. You need to know how many data requests will be issued and how
> much data will have to be retrieved for each request.
> You may need to consider the time distribution of the requests, will
> they all occur at the same time? How long are the users willing to wait
> for their data? What else will be using the line at the same time? Will
> those users be prepared to wait for their data?
>
Thank you for your response. The ODBC access will supplement web based
access, where the web server will be on the same local network as the MySQL
server. The web access will be used mostly for simple queries. The ODBC
access is to be used for more complex ad-hoc queries. So in answer to your
questions, only one person is ever likely to access the mysql database via
ODBC at a time. The reports do not need to be done instantly, so time is not
too much of a worry.
I'm assuming that these are all good answers. What number of users and level
of data did you pass over the 64k link?
Dominic

> Bernard Peek
> London, UK. DBA, Manager, Trainer & Author. Will work for money.
>|||In message <c08skf$b2a$1$8300dec7@.news.demon.co.uk>, Dominic Martin
<dominic.martin@.dplm.co.uk> writes
>"Bernard Peek" <bap@.shrdlu.com> wrote in message
>Thank you for your response. The ODBC access will supplement web based
>access, where the web server will be on the same local network as the MySQL
>server. The web access will be used mostly for simple queries. The ODBC
>access is to be used for more complex ad-hoc queries. So in answer to your
>questions, only one person is ever likely to access the mysql database via
>ODBC at a time. The reports do not need to be done instantly, so time is no
t
>too much of a worry.
>I'm assuming that these are all good answers. What number of users and leve
l
>of data did you pass over the 64k link?
You can work out the data capacity of the link by looking at the
structure of the tables returned and the number of records retrieved.
The number of users isn't important, just the amount of data they use.
That's very difficult to predict with ad hoc reports. Either the person
creating the reports needs to understand the bandwidth they have
available or you have to find a way of saving bandwidth. Have you
considered replicating the mysql database to a local server?
Bernard Peek
London, UK. DBA, Manager, Trainer & Author. Will work for money.|||I have done some work on a database using ODBC over a 64K leased line. The
people paying the bills thought it was "acceptable", but the actual users
thought it abominable. The users didn't find it acceptable until the lines
(one each to three separate user locations) were upgraded from 64KB to T-1
(which is about 1.5MB, as I recall).
A better-written application would have performed better, but the same
people paying the bills wouldn't pay for some obvious improvements, such as
opening the main/first form on a single record rather than on an entire
15,000 record table. But only parts of the app were so poorly
designed/implemented.
Larry Linson
Microsoft Access MVP|||I have done so over ADSL lines (128k upload).
Adhoc queries that returned resonable amounts of data felt OK to me. The
important point is to send the query to the server and have it do the grunt
work - avoiding queries where raw data is returned to Access while it does
the grunt work. However the typical Access form performed substantially
worst - edge of usability.
When you try going long distances or overseas (large latency in the
internet - distance being defined as #routers, hops etc. as opposed to miles
per-se) then things become very bad, because by default Access does not take
care to minimize the amount of round-tripping. So the time is multiple
latency plus data transfer.
"Dominic Martin" <dominic.martin@.dplm.co.uk> wrote in message
news:c08fh8$fmc$1$8300dec7@.news.demon.co.uk...
> Hi,
> I need to setup an ODBC link over our Internet connection between MS
Access
> (local) and a mysql server (Remote - the local PC has the mysql ODBC
driver
> installed). What kind of bandwidth requirements am I looking at? Will it
> function over a 64k leased line? If this would be slow, could there be any
> major impact on other uses such as web and email?
> Regards,
> Dominic
>|||"Bernard Peek" <bap@.shrdlu.com> wrote in message
news:I2QLbrERt$JAFwgF@.shrdlu.com...
> In message <c08skf$b2a$1$8300dec7@.news.demon.co.uk>, Dominic Martin
> <dominic.martin@.dplm.co.uk> writes
I
how
MySQL
your
via
not
level
> You can work out the data capacity of the link by looking at the
> structure of the tables returned and the number of records retrieved.
> The number of users isn't important, just the amount of data they use.
> That's very difficult to predict with ad hoc reports. Either the person
> creating the reports needs to understand the bandwidth they have
> available or you have to find a way of saving bandwidth. Have you
> considered replicating the mysql database to a local server?
>
Yes, but the mysql server is a hosted solution, so we're limited in the ways
that we can get hold of the data. It's either CSV copies of the tables or
the ODBC read only link to the live data. Apart from setting up a SQL server
locally, I'm not sure what else we would need to do. Are you suggesting that
the local SQL server would use ODBC just to make straight copies of the data
or that we would have to download and manually import the CSV files? Under
the latter, it is more likely that we would just import the CSV files
directly into MS Access.
When you say you can work out the data capacity of the link, is there a
rough formula that can be applied. i.e. a certain number of records from a
certain number of tables would require x amount of bandwidth?
Thank you for your continuing help.
Dominic

> --
> Bernard Peek
> London, UK. DBA, Manager, Trainer & Author. Will work for money.
>|||"User" <User@.aol.com> wrote in message
news:OWEP6Y57DHA.2656@.TK2MSFTNGP11.phx.gbl...
> I have done so over ADSL lines (128k upload).
> Adhoc queries that returned resonable amounts of data felt OK to me. The
> important point is to send the query to the server and have it do the
grunt
> work - avoiding queries where raw data is returned to Access while it does
> the grunt work. However the typical Access form performed substantially
> worst - edge of usability.
Thank you for your comments. How do you ensure that the query is sent to the
server to do the work and not have the raw data returned to Access? Is it
something that you have to build into the query?
Dominic

> When you try going long distances or overseas (large latency in the
> internet - distance being defined as #routers, hops etc. as opposed to
miles
> per-se) then things become very bad, because by default Access does not
take
> care to minimize the amount of round-tripping. So the time is multiple
> latency plus data transfer.
>
> "Dominic Martin" <dominic.martin@.dplm.co.uk> wrote in message
> news:c08fh8$fmc$1$8300dec7@.news.demon.co.uk...
> Access
> driver
any
>|||In message <c0a2mq$onm$1$8300dec7@.news.demon.co.uk>, Dominic Martin
<dominic.martin@.dplm.co.uk> writes

>Yes, but the mysql server is a hosted solution, so we're limited in the way
s
>that we can get hold of the data. It's either CSV copies of the tables or
>the ODBC read only link to the live data. Apart from setting up a SQL serve
r
>locally, I'm not sure what else we would need to do. Are you suggesting tha
t
>the local SQL server would use ODBC just to make straight copies of the dat
a
>or that we would have to download and manually import the CSV files? Under
>the latter, it is more likely that we would just import the CSV files
>directly into MS Access.
MySQL supports replication, so that any changes to a master database are
automatically copied to a slave database. You would keep the master
database on the server and run all of your queries against the slave
kept on your own site. You need to talk to your hosting provider about
that, they may have other customers doing the same thing.
If your data changes slowly it may be OK to copy the data to CVS files
once per day and to copy them when the 64k bandwidth isn't being used
for anything else, perhaps overnight. That way you could run queries
against a database that is less than 24 hours old and not need to use
bandwidth during the day.

>When you say you can work out the data capacity of the link, is there a
>rough formula that can be applied. i.e. a certain number of records from a
>certain number of tables would require x amount of bandwidth?
The amount of data to transfer depends on the size of the records. List
the fields in each table and determine the number of bytes required for
each of them. Multiply that by the number of records retrieved. Decide
how long it will take to transfer that data over a 64Kbit (8Kbyte) per
second link. Take account of other people using the same link for other
purposes.
Bernard Peek
London, UK. DBA, Manager, Trainer & Author. Will work for money.

backup/restore strategy help

Hello.

I have only ever been required to take a full back up of my main prod database every night.

Now the times they are-a changing , and it is now required to be able to restore the database up to the last hour.

I've never really done much with tran log / differential backups so I'm asking for some advice as to what should be the best strategy. We are not a 24/7 shop we work from 6:30 am to 6:30 pm every day, so I thought:

1. Full backup @. 7pm

2. Backup tran log every hour after that starting @. 7am (as there are no changes overnight)

How does that sound? also when the tran log is backed up, is it truncated? Or do I need to shrink it?Basically I need to know what to do so it doesn't get too big!

Thanks

That's a good starting strategy.

The tran log will re-use space just the same way it does in simple recovery mode, with one change: It will not re-use space taken up by log records that haven't been backed up yet. Logical when you think about it.

So, with a regular tran log backup job, and a consistent workload, the tran log should reach a steady state, and not need to be shrunk unless it grows unusually due to an extraordinary event, like loading millions of records at once, when that's not the normal pattern.

You can then govern the size of the tran log to a large extent by varying the frequency of your log backups. The more often you back up the logs, the less records are still active due to not having been backed up, so the log doesn't need to grow as large. Also, the more frequently you back up the log, the less work is exposed to loss. Your commitment is to lose no more than an hour, but you can deliver better if you back up more frequently. The flip side of that is that with more frequent log backups, you have more backup files to manage, and restoring becomes more complicated. That's a tradeoff you have to make.

I would also STRONGLY suggest that you do a test restore of a database including log backups so that your procedures for restoring in this configuration are proven. You really don't want to be figuring this out when the database is down and everyone is screaming!