BLOG: Manage the Perfect Storm
In the interface between big-data, data-science, AI and software engineering, a perfect storm is created. There is an ever increasing need for magicians that can control this storm. So what is required of you as developer to be this magician? I will give an overview of some areas that according to my experience are vital to modern data-wizardry.
Power at the tip of your fingers
Infrastructure as code enables you as a developer to create any IT-infrastructure that you need to accomplish your mission, using only a few lines of code. This includes, but is not limited to, computational resources, storage, networks and databases. This also allows you to build solutions that automatically scale or to quickly replicate solutions. It is mind-boggling to realise, that as a developer today you can have all the resources you need, you “only” need to figure out what to do with them.
The data is big and so are the opportunities
During the past four years I have had the opportunity to work with Scania’s now more than 500,000 connected vehicles. The huge amount of data that continuously flows in creates many opportunities for insights about the trucks, the operation of the trucks, the locations where the trucks have an operation, infrastructure needs and in general insights about the transport system. At the same time, working with huge amounts of data requires a different mindset, different methods and different tools.
For example, to truly scale processing of continuous growing huge data sets, you need to work with data streams. Anti-patterns are batch processing of data and methods that rely on vertical scaling of resources.
Just as the magician needs his staff a data-wizard needs his tools. Great tooling for working with data in a scalable way are Hadoop or Snowflake and powerful programming abstractions are SQL or data-frame APIs like Spark.
The power of many
Gandalf worked alone, but this would actually be a mistake if you aspire to become a successful data-wizard. As a team you can form a super brain where you can leverage on your individual strengths and also cover up for your individual shortcomings.
In a diversified team you have the opportunity to approach your challenges from different angles, allowing you to think outside of the box. Also, it becomes easier to gather the varied skill-set that is required to work with advanced analytics.
Last words
In summary, if you are a developer or aspiring to become one, big adventures awaits and there are many facets of your skillset that you can explore. To me big-data, machine-learning, data science, DevOps and software engineering are all different areas that you need insight on as a modern software developer!
