ProfessionalExperience

 

Project1

DevelopmentandSupport

Organization

TechnogridITSystems

 

Description

 

• InvolvingincreatingAzureDataFactorypipelinesthatmove,transform,andanalysedatafroma widevariety ofsources
• TransformthedatatoParquetformatandFilebasedincrementalLoadofdataasperVendorrefreshschedule
• CreatingTriggerstorunpipelinesasperschedule
• ConfiguringADFpipelineparametersandvariable
• CreatepipelinesinParentandchildpattern
• CreatingTriggerstoexecutepipelinessequentially
• MonitoringDremioDataLakeEnginetodeliverdatatothecustomersasperbusinessneeds

 

Project2

DevelopmentandSupport

Organization

TechnogridITSystems

 

Description

 

• CreatedLinkedServicesformultiplesourcesystem(i.e.:Oracle,SQLServer,Teradata,SAPHana,ADLS,BLOB, File StorageandTableStorage).
• CreatedPipeline’stoextractdatafromonpremisessourcesystemstoazureclouddatalakestorage; Extensively worked on copy activities and implemented the copy behaviour’s suchas flatten hierarchy, preserve hierarchy and Merge hierarchy; Implemented Error Handlingconceptthrough copyactivity.
• ExposureonAzureDataFactoryactivitiessuchasLookups,Storedprocedures,ifcondition,foreach,Set Variable,AppendVariable,GetMetadata, Filterand wait.
• Configuredthelogicappstohandleemailnotificationtotheendusersandkeyshareholderswith the help of web services activity; create dynamic pipeline to handle multiple sourceextractingtomultipletargets;extensivelyusedazurekeyvaultstoconfiguretheconnectionsinlinked services.
• Configured and implemented the Azure Data Factory Triggers and scheduled the Pipelines;monitored the scheduled Azure Data Factory pipelines and configured the alerts to getnotificationof failure pipelines.
• Implementeddeltalogic extractionsfor varioussources withthehelpof control table;implemented the Data Frameworks to handle the deadlocks, recovery, logging the data ofpipelines.
• Deployed the codes to multiple environments with the help of CI/CD process and worked oncode defect during the SIT and UAT testing and provide supports to data loads for testing,implementedreusablecomponentsto reducemanual interventions
• KnowledgeonAzureDatabrickstorunSpark-PythonNotebooksthroughADFpipelines.
• Using Databricks utilities called widgets to pass parameters on run time from ADF toDatabricks.
• CreatedTriggers,PowerShellscriptsandtheparameterJSONfilesforthedeployments
• Reviewing individual work on ingesting data into azure data lake and provide feedbacksbasedonreferencearchitecture,namingconventions,guidelinesandbestpractices
• ImplementedEnd-EndloggingframeworksforDatafactorypipelines.
• WorkedextensivelyondifferenttypesoftransformationslikeQuerytransformation,Merge,Validations,Map-operation,HistoryPreservingandTableComparisontransformationsetc.
• ExtensivelyusedETLtoloaddatafromflatfileandalsofrom therelationdatabase.
• Used BODSScriptandGlobalVariables.
• Extracted data from different sources such as Flat files, Oracle to load into SQL database •Ensuring proper Dependencies and Proper running of loads (Incremental and Completeloads).
• Maintained warehouse metadata, naming standards and warehouse standards for futureapplicationdevelopment.
• Involvedinpreparationandexecutionoftheunit,integrationandendtoendtestcases