mysql - PDI or mysqldump to extract data without blocking the database nor getting inconsistent data? -

-

i have etl process run periodically. using kettle (pdi) extract data source database , copy stage database. use several transformations table input , table output steps. however, think inconsistent data if source database modified during process, since way don't snapshot of data. furthermore, don't know if source database blocked. problem if extraction takes minutes (and take them). advantage of pdi can select necessary columns , use timestamps new data.

by other hand, think mysqldump --single-transaction allows me data in consistent way , don't block source database (all tables innodb). disadventage innecessary data.

can use pdi, or need mysqldump?

pd: need read specific tables specific databases, think xtrabackup it's not option.

however, think inconsistent data if source database modified during process, since way don't snapshot of data

i think "table input" step doesn't take account modifications happening when reading. try simple experiment:

take .ktr file single table input , table output. try loading data target table. while in middle of data load, insert few records in source database. find records not read target table. (note tried postgresql db , number of rows read : 1000000)

now question, suggest using pdi since gives more control on data in terms of versioning, sequences, scds , dwbi related activities. pdi makes easier load stage env. rather dumping entire tables.

hope helps :)


Comments

Post a Comment

Popular posts from this blog

html - Firefox flex bug applied to buttons? -

-
i'm trying apply flex button on firefox i'm having problem on chrome doesn't happens. this css: .button { display: inline-flex; align-items: center; } as can see in fiddle, "caret" word moved new line, happens if element <button> tag, <div> works expected. http://codepen.io/anon/pen/xblmgx is firefox bug or there css i'm missing? nevermind, looks "mr firefox" doesn't idea... <button> not implementable (by browsers) in pure css, bit of black box, perspective of css. means don't react in same way e.g. <div> would. this isn't specific flexbox -- e.g. don't render scrollbars if put "overflow:scroll" on button, , don't render table if put "display:table" on it. stepping further, isn't specific <button> . consider <fieldset> , <table> have special rendering behavior: data:text/html,<fieldset style="d
Read more

html - Missing border-right in select on Firefox -

-
Image
when put select div width , padding-left, options have no border on right. more, happend if select have width 118px , parent have 15px of padding left! have idea, going on? my code: <!doctype html> <html> <head lang="en"> </head> <body> <div style="width: 1000px; padding-left: 15px;"> <select style="width: 118px;"> <option>a</option> <option>b</option> <option>c</option> </select> </div> </body> </html> and result: here plunker: click it happend on firefox on windows. tested on 40.0.2 version -webkit-box-sizing: border-box; -moz-box-sizing: border-box; -ms-box-sizing: border-box; box-sizing: border-box; source: https://stackoverflow.com/a/11390432/4431269 source of problem: https://bugzilla.mozilla.org/show_bug.cgi?id=924068 and use of google ! edit: (source of problem, explained) after rea
Read more

c# - two queries in same method -

-
i trying execute 2 queries in same method gives me exception. can red of exception declaring new command there way use same command? string id="hi"; connection.open(); oledbcommand command1 = new oledbcommand(); command1.connection = connection; string query1 = "select * products category='" + combobox1.text + "' , subcategory = '" + combobox2.text + "' , sizes='" + combobox3.text + "'"; command1.commandtext = query1; oledbdatareader reader = command1.executereader(); while (reader.read()) { id = reader[0].tostring(); } textbox1.text = id; string query = "insert category_in (category_id,amount,amount_in) values ('"+ id+"' ,500,300)"; command1.commandtext = query; command1.executenonquery(); messagebox.show("saved"); connection.close();
Read more