Job description

Data Engineering and Delivery @DataWeave

We the Delivery / Data engineering team at DataWeave excel at web scraping, efficiently extracting pricing data from competitor websites and relevant platforms. We are responsible for design and development of robust data collection systems and implement web scraping tools, ensuring the seamless integration of diverse data formats for processing. The delivery team develop data processing pipelines, automate data collection processes, and prioritize data quality assurance, while collaborating closely with other teams to enable effective data analysis required for our pricing dashboards, thus maintaining the company's competitive edge in the market.

Role and responsibilities

In the data operations role within the Data Engineering team, the following roles and responsibilities would typically apply:

1. Data Collection Management: Overseeing the efficient and reliable collection of data from various sources, ensuring the data is gathered in a timely and accurate manner.

2. Data Quality Assurance: Implementing quality control measures to validate and maintain the integrity of the collected data, while identifying and rectifying any inconsistencies or errors.

3. Data Processing and Integration: Managing data processing pipelines and ensuring the smooth integration of different data formats for further analysis and insights.

4. Automation and Tool Management: Developing and managing automation tools to streamline data collection processes, improving efficiency and minimizing manual interventions.

5. Monitoring and Reporting: Monitoring data pipelines and systems to identify potential issues or bottlenecks, providing regular reports to stakeholders regarding data operations performance. 6. Collaboration and Communication: Collaborating with other teams, such as data engineers, product development, and business analytics, to understand their data requirements and ensure the smooth flow of data operations within the company.

7. Process Improvement: Continuously identifying opportunities for process improvement, optimizing data operations to enhance efficiency and overall performance.

8. Documentation: Maintaining comprehensive documentation of data operations processes and procedures, facilitating knowledge sharing and ensuring continuity within the team.

Mandatory skills required

• Proficiency in scripting languages Python (sufficient knowledge of Pandas / NumPy / Scrapy and other libraries that would help in data operations)
• Familiarity with API integration and usage to collect data from various sources
• Familiarity with documentation tools such as Confluence, Jira
• Familiarity with troubleshooting data-related issues and implementing effective solutions.
• Ability to use debugging tools and techniques to resolve data operation challenges
• Web scrapping nice to have