Resume
My professional journey and expertise
Vijay Anand Pandian
Senior Data & AI Engineer
London, United Kingdom
Skills
About Me
I'm Vijay Anand Pandian, a seasoned Data & AI Engineer with over 10 years of experience solving complex data problems, building Big Data pipelines, and deploying Machine Learning models across major cloud platforms including Azure, AWS, and GCP.
Areas of Expertise
Domain Knowledge
Sales, Marketing, Product Analytics, Cyber Security
Core Interests
Big Data pipelines, DevOps, MLOPS
Programming Languages
Python, Scala, Java
Big Data Technologies
Hadoop, Spark, Hive, Databricks
Database Systems
Postgres, Oracle, MS-SQL, MongoDB, Elasticsearch
Data Warehouses
Snowflake, Redshift, Hive, BigQuery
Web Technologies
Django, Flask, React
Cloud Platforms
AWS, GCP, Azure
Certifications
Apache Airflow Fundamentals
Apache Airflow DAG Authoring
Professional Experience
Data Engineer | Sky UK
Dec 2022 – Current | London, United Kingdom
- AWS S3 (Parquet) files batch load to spectrum tables and create a star-schema tables and views top of it.
- Smart solutions on cloud monitoring
Data Engineer | Channel4
May 2022 – Dec 2022 | London, United Kingdom
- AWS S3 (Parquet) files batch load to spectrum tables and create a star-schema tables and views top of it.
- Smart solutions on cloud monitoring (martech, advertisement, targetted audience)
Data Engineer | Eli Lilly and Company
May 2021 – April 2022 | Bangalore, India
- Re-architected legacy queue application with cutting edge technologies using IBM MQ, AWS - MQ,Lambda,DynamoDB,Fargate.
- Built Data pipelines using Azure Databricks to push data to Data Lake.
Software Engineer | Data Engineer | United Health Group ‑ Optum
July 2019 – April 2021 | Chennai, India
- Worked as a full stack developer. Develop and enhance the product features listed for GA. Worked in backend, frontend and infrastructure tasks.
- Migrated the data from Oracle DB to AWS S3 using Spark, Scala and EMR.
- Automated the data quality checks. Converted complex Oracle SQL to Spark SQL.
- Built ETL data pipeline to extract data from Azure Blob to AWS S3 using rclone and pulsar. Used Airflow for workflow management and Terraform for cluster creation.
Data Analytics Engineer | Gartner
September 2015 ‑ July 2019 | Chennai, India
- Built Email Bot product that extracts data from outlook mail box and generates sales leads and missing clients info based on auto reply emails using python and MongoDB
- Built data pipelines for a marketing and insights team to extract google analytics data from GCP BigQuery to AWS S3.
- Designed competitor analytics dashboard for events organizing team to understand the competitor past and future event details using python, Elastic Search and Kibana
- Migrated data from Oracle to Datalake using Sqoop. Orchestrated workflow with Apache Airflow. Initiated CICD pipeline in AWS to deploy machine learning models
Associate Software Engineer | Symantec ‑ Norton
Jan 2014 ‑ Aug 2015 | Chennai, India
- Built automation framework using Python for Norton Antivirus for Mac product to test the various security scenarios.
- Contributed and maintained the common libraries used for different team usages.
Education
Amrita Vishwa Vidyapeetham
Master's in Cyber Security
July 2014 | Coimbatore, TN, India
University College of Engineering Tindivanam
Bachelor's in Information Technology
June 2012 | Tindivanam, TN, India
Professional Journey
I started my journey with a Bachelor's in Information Technology followed by a Master's in Cyber Security. While pursuing my Master's, I joined Symantec as an intern where I worked with the Managed Security Services team on their SIEM tool. This experience taught me about data architecture including network devices, servers, and large data flows.
After my internship, I moved to the Norton Product Team where I developed Python automation frameworks for security testing. Later at Gartner, I evolved into a data analytics professional, creating solutions like:
- Email Bot - A classification system for managing sales contacts
- Competitive Dashboard - A tool for market analysis using web scraping
- Database Cleaner - A machine learning tool for data cleansing
- ETL Solutions - Moving data between various platforms
At Optum, I took on the dual role of a Product Owner and Full Stack Engineer, managing all aspects of the Initiative Manager product. I also worked as a Data Engineer, transforming complex Oracle queries to Spark.
My most recent role at Eli Lilly involved modernizing legacy applications using cloud technologies and developing robust Python frameworks.