Data engineering

Data engineering

In recent years, there have been huge developments in the field of data engineering. Especially larger corporations have adopted complex big data systems and analytical machine learning methods in order to improve their operational effectiveness and quality of their business intelligence. They have increased the performance of processes such as demand forecasting, production scheduling, transportation routing and service logistics.

Smaller enterprises can equally benefit from these developments, however due to high costs and lack of specialized in-house knowledge, they have been restrained in adopting data engineering solutions. We have identified four competence fields necessary for successful data engineering and found out that these fields have recently become much more accessible:

  1. Technology to collect data: Due to the developments in mobile technology and Internet of Things, equipment to measure data has become much more affordable and network interactive. Also the amount of Open Data has grown exponentially. 90% of all data ever recorded is less than 2 years old.
  2. Processing power of computers: Ongoing developments in the performance of computers enable us to perform elaborate operations on smaller machines, making it more affordable. Nowadays, simple data mining algorithms can run on personal computers and laptops.
  3. Academic knowledge: In recent years, the methods to apply statistical analysis and machine learning have been formalized and standardized, making it more accessible and easier to apply in a business context.
  4. Business know-how: Due to the pilots of academic institutions and multinationals, a solid knowledge base has developed towards the practical possibilities and limitations of data engineering.

At Phinion, we have all the competences to capitalize on these developments. We use this to provide valuable, cost effective investments in data engineering solutions. We provide these four fields of data engineering:

  1. Collect: The software and hardware to measure data directly or collect it from IT systems or open internet sources.
  2. Organize: The IT competences to provide the software and hardware to store and process the data.
  3. Analyze: We have the statistical and mathematical knowledge and academic network to build the needed algorithms.
  4. Utilize: We have the business know-how to understand your needs and the knowledge of what is realistically possible.

DataEngineering


 
 

Opportunities of data engineering

Data engineering solutions are useful for improving any operation that is data driven. We are able to create better quality information by using more data and using better statistical methods. This way, we improve processes such as Demand Forecasting, Quality Control, Statistical Process Control and Human Resource Scheduling.

The drawbacks of spreadsheets

Initially, spreadsheets are useful for basic data engineering applications, however they have their limitations. They are only able to handle a restricted amount of data and their performance scales poorly as the complexity and size of a calculation increases. It also takes a lot of effort to standardize the use of spreadsheets consistently within an organization. Usually everyone has their own formats, which are difficult to modify.

Improve scalability

The first direct improvement data engineering solutions provide towards spreadsheet implementations is the ability to perform analysis on much larger data sets. Exactly the same calculations can be done, but with much more data and factors on 10 faster. Calculation times can literally go down from hours to minutes.

Integrate the analysis process

Furthermore, our data engineering solutions can be connected to existing software and hardware systems to retrieve data and export results. This eliminates a lot of manual work and assures fast feedback.

Automate calculations

This integration allows the analysis to be done automated and continuous. The solution is able to perform real time monitoring and update the analysis as soon as changes to the data are made, for example through notifications and push messages. This enables the company to sense and respond in an agile matter.

Increase intelligence

Finally, we can extend the depth and quality of the analysis by adding machine learning methods that uses data mining to recognize new patterns and learn new correlations. Such methods can enrich the analysis processes, improve their quality and accuracy and offer new insights.


 
 

Collect

Phinion realizes the hardware and software to collect data. We design and built integrated solutions tailored to specific customer context and needs. We test, install and deliver operational ready-to-use solutions.

We deliver solutions to collect data in various situations:

  • Existing IT systems: We built software to retrieve data from ERP, CRM and accounting systems. Whenever possible we use existing export functionalities. Our solutions are capable to extract this data automatically and real time.
  • Smart devices: We create the software and hardware to extract data from devices, such as Smart Phones, GPS trackers, production machines, vehicles and CCTVs. Also for these devices we use existing export functions and we realize automatic and real time solutions.
  • Open data: This is public data shared by for example government and academic institutions and for example sales and forecast data shared by big customer organizations. Also for this data we built the necessary interfaces which ensure security and privacy whenever necessary.
  • New measure systems: When required, we built combined hardware and software solutions to measure processes, for example bar code and RFID systems. We also realize the technical integration of these systems into existing IT infrastructure.


 
 

Organize

Phinion realizes the hardware and software to organize data and make it accessible in the right format. We create the necessary database solutions and interface applications to make data available for users and IT systems. Whenever necessary, we install and configure the required hardware for data storage and communication. We also provide cloud solutions for data storage and processing.
Our solutions offer the following possibilities:

  • Structure: We are able to combine data from various sources and restructure it. This allows us to enrich the value of retrievable information.
  • Aggregation: We extract information from the data in any desired shape and level of detail. These custom exports allow multi-purpose applications of the data.
  • Data cleaning: We are able to automatically detect gaps and errors in the data. Especially when data sources are combined, the detection of inconsistencies and their resolution is valuable.
  • Interoperability: Our solutions are able to realize a real time connection between IT systems, allowing continuous synchronization.


 
 

Analyze

Phinion develops the software to perform intelligent analysis to large amounts of data. This software is built to do this continuously, realizing up to date results. The algorithms to do this are custom made to specifications.

Our algorithms can be designed with learning capabilities. This allows them to be designed to grow along with expected organizational developments. The flexible design allows them to be easily adapted whenever necessary.

We develop three types of algorithms listed here in ascending complexity and capability:

  • Descriptive: These algorithms produce factual information, the so called “As-Is” figures. We design these algorithms to specification, allowing information to be provided in every desired format and detail level.
  • Predictive: These algorithms provide information of a forecasting nature. This is done by calculating correlations and fit statistical distributions to patterns in the data.
  • Machine learning: These algorithms explore the data in order to find patterns. This process is known as data mining. The algorithm remembers the patterns found and learns the regularities in the data.


 
 

Utilize

Phinion realizes the software and hardware solutions to use analysis results. These solutions are integrated into existing systems and delivered ready to use. This way we assure the direct value of our data engineering solutions.
We deliver the following solutions:

  • Reporting: We built interactive user interfaces with custom dashboards. We can show the standard figures (bar charts, line charts, etc), but also more sophisticated visualizations, such as geographical maps and process lay-outs. These dashboards are useful for tactical/strategic decision making, but also to realize real time dashboards for direct operational feedback and control.
  • System feedback: We develop software interfaces to communicate with existing hardware and software. These use the analysis results directly for operations control. We can realize real time feedback to both software systems for planning and scheduling and hardware systems, such as production machines.