Falcon Smart IT (FalconSmartIT)’s Post

DataStage - Run Time Column Propagation. While I am developing DataStage jobs I came across a situation which surprise me and took some time to figure it out What was the issue. So We have source file and reference file which joins through look-up stage. At Look-up Stage we dropped few columns and load it into target file. surprisingly We see dropped Columns also even we were not configured. Reason is when we check "Run Time Column Propagation" option weather you required or not those columns will flow towards target. So Keep on eye on Run Time Column Propagation option when you are dropping columns.

🔍 Looking for IT solutions? Look no further! Get in touch with us to explore staffing options, business continuity planning, cloud computing, DevOps engineering, and more! 💻 Our team of experts includes IBM DataStage and Informatica developers, Java and .Net developers, and many others! 🌐 #informationtechnology #staffing #businesscontinuity #cloudcomputing #devops #ibmdatastage #informatica #java #dotnet #technology #itexperts #computerscience #digitaltransformation #innovation #techsolutions #ustringsolutions 🚀

Automate History / Day 0 Load I have developed a DataStage job which extract data from Teradata and put it into Parquet file. But it required some manual efforts to extract 300+ tables. So thought of creating a Shell Script which take input of Config/text file which contain List of table names, Key Column to Split export, Total Splits, Split. So Shell Script scan all lines from Config/text file and run the DataStage job (which do history load) with different instance. But there is a problem I see in DataStage. in development environment one DataStage job not triggeing more then 4 instance So I wrote Script should not run not more then 4 instaces. Please find Script below.

ETL engineer Onsite in Katy, TX and in the office daily. Houston TX might be a possibility.   Role Overview & Required Skills: Targeting candidates focused on Data Acquisition and Data Integration using DataStage. The contractor should have deep experience working with flat files, XML transformations, as well as ETL development using DataStage V8 or greater. They need to have solid experience with Quality Stage Address Standardization CASS modules. The contractor needs experience integrating the components of ORACLE, SQL, and DataStage (ETL). Excellent communication skills are a must.   Responsibilities: 1. Data Extraction: Extract data from various sources such as databases, files, APIs, and web services using ETL tools or programming languages. 2. Data Transformation: Cleanse, validate, and transform the extracted data to ensure its accuracy, consistency, and integrity. This may involve data mapping, data conversion, data aggregation, and data enrichment. 3. Data Loading: Load the transformed data into the target systems such as data warehouses, data marts, or operational databases. This includes defining data structures, creating tables, and optimizing data loading processes. 4. ETL Process Development: Design, develop, and maintain ETL processes and workflows using ETL tools (e.g., Informatica, Talend, SSIS) or programming languages (e.g., Python, SQL). This involves writing efficient and scalable code to handle large volumes of data. 5. Data Quality Assurance: Perform data quality checks and implement data validation rules to ensure the accuracy, completeness, and consistency of the data. Identify and resolve data quality issues or anomalies. 6. Performance Optimization: Optimize ETL processes for improved performance, scalability, and efficiency. Identify and resolve performance bottlenecks, optimize data transformations, and fine-tune data loading processes. 7. Documentation and Reporting: Document ETL processes, data mappings, and data lineage for future reference. Generate reports and provide insights on data quality, data lineage, and ETL process performance. 8. Collaboration and Communication: Collaborate with cross-functional teams including data analysts, data engineers, business analysts, and stakeholders to understand data requirements, gather feedback, and ensure successful ETL implementation.   Requirements: