Tuesday, March 27, 2012

Basic logging/processing db question

I have many worker systems logging data to a central database.
The central database needs to periodically process all that data. During this processing time, new records can't be added part way through and the workers still need to log the data somewhere.

I was thinking of the following approach:

1) Rename the table the worker logs to. Let's say from LogTable to ProcessingTable
2) Immediately create a new and empty table for the worker to log to (LogTable). Possibly wrap this and the preceeding rename step in a transaction which may provide some atomicity.
3) Process the ProcessingTable
4) Drop ProcessingTable.

To the db experts here, does this sound like a decent approach or are there problems with this?Sounds a bit dirty
can you give a bit more info on the type of processing. is this cube population or just calculations for oltp?
how often? and how much(data)?

I would prefer to put the new data in table A
use dts to process the data every hour and place it in the destination table b
and have querying happen full time on table b

this way you only have 1 hour of latency in your data
if the info is not as volitile then schedule processing more closely together.|||Problem #1 - you haven't paid the bill. Do you need an official invoice?
Problem #2 - you can't count on free advice while supporting your production.
Problem #3 - when advice is given and it's proven to be the right solution, - you need to follow it rather than getting yourself into unchartered waters with bigger problems on your hands.

Is that enough?|||sounds to me like someone is preaching the brainwashing he just got at his "Fanatical Customer Service" Based Job :p|||No, Scott, this has nothing to do with "Fanatical" brainwashing, it has to do rather with the guy paying the bill for the services that he received from me. Rings the bell now?|||oh what was i thinking poking fun at you? :rolleyes:

i'm just full of sour grapes because they ..... well ill save that story for next week when i get back and we get drunk. :D|||Hey, can't wait!!!

Kurt,

It's really not difficult, just dial the number and say that you guys were VERY busy, and the hole thing will go away, you know?!sql

No comments:

Post a Comment