[xrecnet] Bigdata - Spark Developer - Architect - Remote

Hi

 

This is Aravind, - Recruitment Team from MSR Cosmos

We have an urgent requirement as follows:

Please respond with resumes in MS-Word Format with the following details to aravind@msrcosmos.com

Full Name :
Location :

Relocation :
Contact Number :
Email :

Skype Id :

Last 4 digit SSNO :
Availability for project :

Availability for Interviews :
Visa Status and Validity :

D O B :

Years of Exp :

Requirement Details

 

Number of resources & roles required	1 - Solution Architect
Duration (number of days/hours required)	35 Days
Can Project be delivered by a Partner resource?	Yes
Expected Start date	11th October
Timezone	EST
Onsite or Remote? If onsite, state location	Remote
Skillsets/Platform experience required	Spark, Spark Streaming
Project/scope description	Main qualification is strong Spark and Spark Streaming development experience. NiFi development experience is a plus. This is not a sys admin position. Please focus on App Developers.  1.0 Create a JMS to Kafka Custom Data flow connector using Spark/Scala 1.1 Development 1.2 Unit testing 2.0 Nifi to Spark: BOS CAD Use Case 2.1 Refactor Avro Schema Registry 2.2 Assist with Kafka topics, migration from HDF to CDP and pushing data to Kafka topics 2.3 Complete Nifi to Spark conversion tasks, including: o Use JMS to Kafka connector to publish the data to Kafka o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish the parsed & processed data to Kafka o Consume from Kafka & write to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 2.4 Complete User Testing and Resolution Activities (Data and Functional) 2.5 Complete all deployment and migration activities, including HDF to CDP Migration tasks 2.6 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 2.7 Provide Knowledge Transfer 3.0 Nifi to Spark: BOS LOCO & File Transfer Use Case 3.1 Refactor Avro Schema Registry 3.2 Assist with Kafka topics, migration from HDF to CDP and pushing data to Kafka topics 3.3 Complete Nifi to Spark conversion tasks, including: o Use JMS to Kafka connector to publish the data to Kafka o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish the parsed & processed data to Kafka o Consume from Kafka & write to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 3.4 Complete User Testing and Resolution Activities (Data and Functional) 3.5 Complete all deployment and migration activities, including HDF to CDP Migration tasks 3.6 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 3.7 Provide Knowledge Transfer 4.0 Nifi to Spark: UTCS Use Case 4.1 Complete Nifi to Spark conversion tasks, including: o Pull Data from Oracle DB for following sub-categories [total 132 tables] ■ Blast & Reload ■ Append Only ■ Merge o Parsing, processing, transformations o Publish the data to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts o Capture Audit information ■ Capture start and end timestamp for each table processing. Capture the counts for each table ■ Use the existing audit tables for storing audit data [refactor to simplify the design] 4.2 Complete User Testing and Resolution Activities (Data and Functional) 4.3 Complete all deployment and migration activities, including HDF to CDP Migration tasks 4.4 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 4.5 Provide Knowledge Transfer 5.0 Nifi to Spark: ITCM Use Case 5.1 Refactor Avro Schema Registry 5.2 Assist with Kafka topics, migration from HDF to CDP and pushing data to Kafka topics 5.3 Complete Nifi to Spark conversion tasks, including: o Ingest the logs data from NFS mount(s) and publish to Kafka o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Write the data to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts o Capture Audit Information ■ Capture timestamp (start/end) for each table ■ Capture counts for each table ■ Refactor existing audit tables for audit data storage 5.4 Complete User Testing and Resolution Activities (Data and Functional) 5.5 Complete all deployment and migration activities, including HDF to CDP Migration tasks 5.6 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 5.7 Provide Knowledge Transfer 6.0 Nifi to Spark: ITCSM Use Case 6.1 Refactor Avro Schema Registry 6.2 Assist with Kafka topics, migration from HDF to CDP and pushing data to Kafka topics 6.3 Complete Nifi to Spark conversion tasks, including: o Use JMS to Kafka connector to publish the data to Kafka o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish the parsed & processed data to Kafka [Staging copy] o Consume from Kafka[Staging copy] & write to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 6.4 Complete User Testing and Resolution Activities (Data and Functional) 6.5 Complete all deployment and migration activities, including HDF to CDP Migration tasks 6.6 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 6.7 Provide Knowledge Transfer 7.0 Nifi to Spark: PTC Trip Session 7.1 Refactor Avro Schema Registry 7.2 Complete Nifi to Spark conversion tasks, including: o Consume from Kafka[Staging copy] o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish data to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 7.3 Complete User Testing and Resolution Activities (Data and Functional) 7.4 Complete all deployment and migration activities, including HDF to CDP Migration tasks 7.5 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 7.6 Provide Knowledge Transfer 8.0 Nifi to Spark: PTC District Session 8.1 Refactor Avro Schema Registry 8.2 Complete Nifi to Spark conversion tasks, including: o Consume from Kafka[Staging copy] o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish data to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 8.3 Complete User Testing and Resolution Activities (Data and Functional) 8.4 Complete all deployment and migration activities, including HDF to CDP Migration tasks 8.5 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 8.6 Provide Knowledge Transfer 9.0 Nifi to Spark: PTC Locomotive Position 9.1 Refactor Avro Schema Registry 9.2 Complete Nifi to Spark conversion tasks, including: o Consume from Kafka[Staging copy] o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish data to Kudu o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 9.3 Complete User Testing and Resolution Activities (Data and Functional) 9.4 Complete all deployment and migration activities, including HDF to CDP Migration tasks 9.5 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 9.6 Provide Knowledge Transfer 10.0 Nifi to Spark: Cassandra 10.1 Refactor Avro Schema Registry 10.2 Complete Nifi to Spark conversion tasks, including: o Consume from Kafka o Schema registry integration o Schema parsing & validation o Parsing, processing, transformations o Publish data to Cassandra o Exactly-Once processing o Exception handling o Wrapper Script/Metadata Files o Auto restarts 10.3 Complete User Testing and Resolution Activities (Data and Functional) 10.4 Complete all deployment and migration activities, including HDF to CDP Migration tasks 10.5 Create and finalize Documentation (including Mapping Documents and Test Artifacts) 10.6 Provide Knowledge Transfer 11.0 Performance Metrics and Monitoring Jobs METRICS 11.1 Record count audit o Move to a shell script from current NiFI flow which approximates Kafka vs. Hive counts 11.2 Latency audit o Add audit columns to Header & select child to Ten (10) Kudu tables o Add Kafka to Kudu tables o Fetch differential 11.3 Cassandra Tables o Add Kafka to Three (3) Cassandra Tables o Fetch the differential to identify Kafka to Cassandra latency 11.4 Latency Query o Create query to capture latency ■ Parameterized to accept DB & Tables ■ Exclude Cassandra Tables 11.5 Query performance o Refactor to adjust to Impala & Kudu. Remain as a shell ■ Parameterized to accept DB & Tables MONITORING 11.6 Kafka alerts o Producer Activity ■ Milliseconds lapsed since producer was active o Consumer Activity ■ Milliseconds lapsed since producer was active o Consumer Group Latency ■ Application dependent o End to End Latency 11.7 Topic Partition Bytes Alert configuration o Alert-1: Value = 0, creates alert when the topic partition becomes idle o Alert-2: Value > max_bytes_in_expected, creates an alert when the topic partition input load is higher than usual
Any other comments

 

 

Stay Safe

 

Thanks & Regards, 

Aravind
MSR COSMOS

6200 Stoneridge Mall Rd, Ste 300, Pleasanton, CA - 94588

Desk : 925 399 7145

Textnow : 732 574 5974

Fax : 925-219-0934

Email : aravind@msrcosmos.com   
URL: http://www.msrcosmos.com

-          Microsoft Gold Partner

-          SAP Silver Partner

-          Oracle Gold Partner

-          Hortonworks Silver Partner

-          Cloudera Silver Partner

-          E-Verified

-          WBE Certified

 

 

Note: This email is not intended to be a solicitation.  Please accept our apologies and reply in the subject heading with REMOVE to be removed from our Mailing list.

 

Any information and documentations including Govt issued ID or government issued documents if forged, manipulated or Falsified is considered as a felony as per US laws Such actions are a punishable offense and can lead to Criminal Investigation or Indictment

 

Confidentiality Notice: 

Unless otherwise indicated, email transmission is not a secure form of communication and your reply may not be encrypted. The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately.

 

--
You received this message because you are subscribed to the Google Groups "Xrecnet IT Recruiters Network - Corp to Corp IT Jobs & Hotlists" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xrecnet+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xrecnet/CAMSx0H0Z5xoExJfEUgAS4T2oreBW8XqV0H2%2Br5oOCojBw_H3yw%40mail.gmail.com.